Streaming Data and Historical Data

[SirNewton 3]

Hello @Physicsman

While processing Historical data, I saw something unusual that I need your help with.

We know that the Steaming Data along with the market closes on FRIDAY 9:59PM GMT

And, we know that the Steaming Data along with the market opens on SUNDAY 10:00PM GMT

And, we know that Each "day" historical data covers from midnight (00:00) to 23:59 GMT

But, I noticed that in the Historical Data that was outside that range. See Blow:

AUD/JPY:
ASK: {"BarDate": "2021-02-20 00:00:00", "Open": 83.062, "High": 83.062, "Low": 83.062, "Close": 83.062}
BID: {"BarDate": "2021-02-20 00:00:00", "Open": 82.912, "High": 82.912, "Low": 82.912, "Close": 82.912}

USD/CHF:
ASK: {"BarDate": "2021-02-20 00:00:00", "Open": 0.89672, "High": 0.89672, "Low": 0.89672, "Close": 0.89672}
BID: {"BarDate": "2021-02-20 00:00:00", "Open": 0.89607, "High": 0.89607, "Low": 0.89607, "Close": 0.89607}

USD/JPY:
ASK: {"BarDate": "2021-02-20 00:00:00", "Open": 105.514, "High": 105.514, "Low": 105.514, "Close": 105.514}
BID: {"BarDate": "2021-02-20 00:00:00", "Open": 105.392, "High": 105.392, "Low": 105.392, "Close": 105.392}

2021-02-20 is a Saturday. The other pairs, I am processing had a BarDate of "2021-02-19 00:00:00", which is what I expected.

Can you please explain the discrepancy of the BarDate of pairs AUD/JPY, USD/CHF, and USD/JPY?

Thank you.

[Physicsman 26]

Hi SirNewton,

I'll look into this for you and reply back with our findings.

Kind Regards, PM

[Physicsman 26]

Hi SirNewton,

According to the Pricing team, an erroneous price tick was sent after market close on Friday resulting in the Saturday price bar. The team are working on removing this from the chart database.

Kind Regards, PM 

[SirNewton 3]

Thank you. I looked through the full history and found similar out of bound entries. Will provide sqlscript used to find out of bound entries.

Note: I use Postgres, but you should be able to do similar queries in other databases. In the DB datetime is "BarDate".

Will give you Saturday and Sunday

select * forex_history where extract(isodow from datetime)in(0,6,7) 

Or to see just Saturday

select * forex_history where extract(isodow from datetime)=6

[SirNewton 3]

"According to the Pricing team, an erroneous price tick was sent after market close on Friday resulting in the Saturday price bar"

By the way, if those price ticks are actual and not generated by error. I rather you keep it in the history. I can write code to combine those with Friday historical.

[Physicsman 26]

Hi Sir Newton,

If those are indeed actual price ticks the team will be able to correct it appropriately without losing the data. 

Kind Regards, PM