Bug with reading features from json

We have built a website which takes features with attributes from a database, reconstructs the features using pyGPlates on the server and returns them to a map and as json.

We bumped into a bug. If an attribute of the first data point has a value null this attribute is ignored for all features, even if those do have a non-null value. We can implement a work around (replace null with a dummy), but this behaviour is unwanted. I can send test code, with sample inputs of different cases.

Using revision 28 on Ubuntu 16.04

This is the code:

import pygplates
import json
import geopandas as gpd

tests = ['', '_with_all_nulls', '_with_all_data', '_with_null_and_data', '_with_data_and_nulls']

model = './test_in/Poblete_MasterBlocks_V4.0.shp'
age = 0

def reconstruct(model, age, feature_path):
    """ Reconstruction of geometries on base of given dataset with assigned plate_ids
    features = pygplates.FeatureCollection(feature_path)
    rot_file = './test_in/FPG_master_AHS_v4.0.rot'
    abs_path = f'./test_out/reconstructed{test}.shp'
    pygplates.reconstruct(features, rot_file, abs_path, age)
    geo_dict = json.loads(gpd.read_file(abs_path).to_json())
    return geo_dict

for test in tests:
    abs_path = f'./test_in/response_db{test}.json'

    geodict = reconstruct(model=model, age=age, feature_path=abs_path)

Hi t.j.m.vanderlinden,

I have received the test files in my email. It seems that the problem happens when pygplates saves the reconstructed data to a “.shp” file.

Using the following code, you can see the ‘_18_o’ has been loaded into the “feature”.

    for feature in features:

If you save the reconstructed geometries to a GMT xy file, you can see the ‘_18_o’ has been preserved.
abs_path = f'./test_out/reconstructed{test}.xy'

The column ‘_18_o’ gets lost when saving to “.shp” and ORG GMT file format. I will talk to @john.cannon to see what we can do about it.

Thanks Michael!
Hope you’ll find a solution.
It would be nice if there is an option to export directly as geojson too.

For future reference: the issue happens on the export from pygplates to shape file or OGR GMT file. Export to GMT xy preserves the attributes.

Hi Thomas,

Thanks for the test files and code to reproduce the issue. That helped Michael and I to locate the issue.

As Michael noted, the problem happens when saving to an OGR format file (eg, Shapefile, OGRGMT) but the attributes are correctly loaded when reading an OGR file (before saving).

This affects pyGPlates and GPlates alike.

I just fixed it for saving but not yet exporting (which is the code path that pygplates.reconstruct() uses). Once that’s done I’ll PM you the source code (to compile on Ubuntu and test).


For reference: It turned out that the attribute field names were collected from the first feature in the collection, which explains why your ‘_with_null_and_data’ file has problems (ie, it’s first feature has a null, followed by three non-nulls). Another thing which may or may not affect things is Ubuntu 16.04 uses GDAL < 2.2. With GDAL >= 2.2 we can read/write null properly. In other words Ubuntu 18.04 (GDAL 2.2) handles null better - but I’m not sure if this will affect your issue. So just flagging it for reference.

1 Like

For reference:
I found a solution in adding a dummy feature before all others with all attributes having a value and removing this dummy feature after reconstruction.