Skip to content

API outputFormat=parquet throws exception on TimeScaleType #595

Description

@runejo

When serialising to Parquet the https://github.com/PxTools/PCAxis.Serializers will on some tables fail with ArgumentException("Invalid TimeScaleType or value."). The reason is that both PX file format and Paxiom only support a given set of timescales. When custom timescales are defined in CNMM they will be classified in Paxiom as TimeScaleType=NotSet or they will have a valid TimeScaleType but with invalid codes.

Examples

curl -s "https://data.qa.ssb.no/api/pxwebapi/v2/tables/07314?lang=en" | jq | grep timeUnit
  "timeUnit": "Other",

curl -is "https://data.qa.ssb.no/api/pxwebapi/v2/tables/07314/data?lang=en&outputFormat=parquet" | grep HTTP
HTTP/2 500 
curl -s "https://data.qa.ssb.no/api/pxwebapi/v2/tables/04555?lang=en" | jq | grep timeUnit
  "timeUnit": "Annual",

curl -is "https://data.qa.ssb.no/api/pxwebapi/v2/tables/04555/data?lang=no&outputFormat=parquet&valuecodes%5BContentsCode%5D=Personer&valuecodes%5BTid%5D=2006,2006%20Gml&valuecodes%5BKjonn%5D=*&valuecodes%5BArbledUndeSyss%5D=*" | grep HTTP
HTTP/2 500

curl -s "https://data.qa.ssb.no/api/pxwebapi/v2/tables/04555/data?lang=no&outputFormat=px&valuecodes%5BContentsCode%5D=Personer&valuecodes%5BTid%5D=2006,2006%20Gml&valuecodes%5BKjonn%5D=*&valuecodes%5BArbledUndeSyss%5D=*" | grep TIMEVAL
TIMEVAL("?r")=TLIST(A1),"2006","2006 Gml";

Possible solution

The reason we look at TimeScaleType is that we wan't to make a timestamp column i parquet with at valid DataTime. One solution would be to make the timestamp column optional and only populate the column when we can produce a DataTime

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    • Status
      Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions