Skip to content

[DCP - SigFigs] Preserve SigFigs through Dataflow. Stop converting to double/int #723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gmechali wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[DCP - SigFigs] Preserve SigFigs through Dataflow. Stop converting to double/int #723

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
40ee4f5
Preserve SigFigs through Dataflow. Stop converting to double/int
gmechali Aug 14, 2026
4d81044
Remove the MCF obs number bc we removed all usages of it. Reserve the…
gmechali Aug 14, 2026
c40baa0
Report updates
gmechali Aug 14, 2026
1c61b07
Statcheck optimization
gmechali Aug 14, 2026
1e7a534
StatChecker code cleanup of if branch
gmechali Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 1 addition & 7 deletions pipeline/util/src/main/java/org/datacommons/ingestion/util/GraphReader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -495,13 +495,7 @@ static TimeSeriesKey toTimeSeriesKey(McfStatVarObsSeries.Key key, String importN
}

static Observation toObservation(TimeSeriesKey seriesKey, StatVarObs obs) {
String value = "";
if (obs.hasNumber()) {
value = Double.toString(obs.getNumber());
} else if (obs.hasText()) {
value = obs.getText();
}

String value = obs.hasText() ? obs.getText() : "";
return Observation.builder().seriesKey(seriesKey).date(obs.getDate()).value(value).build();
}

Expand Down
22 changes: 21 additions & 1 deletion pipeline/util/src/test/java/org/datacommons/ingestion/util/GraphReaderTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,21 +386,41 @@ public void testToObservation() {
"testStatVar", "geoId/testPlace", "", "P1Y", "testMethod", "testUnit", "100", facetId);

StatVarObs obs1 =
StatVarObs.newBuilder().setDcid("obs1").setDate("2020").setNumber(10.0).build();
StatVarObs.newBuilder().setDcid("obs1").setDate("2020").setText("10.0").build();
StatVarObs obs2 =
StatVarObs.newBuilder().setDcid("obs2").setDate("2021").setText("someText").build();
StatVarObs obs3 =
StatVarObs.newBuilder().setDcid("obs3").setDate("2022").setText("12.5000").build();
StatVarObs obs4 =
StatVarObs.newBuilder()
.setDcid("obs4")
.setDate("2023")
.setText("100000000000000001")
.build();

Observation expected1 =
Observation.builder().seriesKey(seriesKey).date("2020").value("10.0").build();

Observation expected2 =
Observation.builder().seriesKey(seriesKey).date("2021").value("someText").build();

Observation expected3 =
Observation.builder().seriesKey(seriesKey).date("2022").value("12.5000").build();

Observation expected4 =
Observation.builder().seriesKey(seriesKey).date("2023").value("100000000000000001").build();

Observation actual1 = GraphReader.toObservation(seriesKey, obs1);
assertEquals(expected1, actual1);

Observation actual2 = GraphReader.toObservation(seriesKey, obs2);
assertEquals(expected2, actual2);

Observation actual3 = GraphReader.toObservation(seriesKey, obs3);
assertEquals(expected3, actual3);

Observation actual4 = GraphReader.toObservation(seriesKey, obs4);
assertEquals(expected4, actual4);
}

@Test
Expand Down
40 changes: 35 additions & 5 deletions pipeline/util/src/test/java/org/datacommons/ingestion/util/PipelineUtilsTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,10 @@ private McfGraph createStatVarObservationGraph(
return graph.build();
}

private McfStatVarObsSeries.StatVarObs createStatVarObs(String date, double value, String dcid) {
private McfStatVarObsSeries.StatVarObs createStatVarObs(String date, String value, String dcid) {
McfStatVarObsSeries.StatVarObs.Builder svObs = McfStatVarObsSeries.StatVarObs.newBuilder();
svObs.setDate(date);
svObs.setNumber(value);
svObs.setText(value);
svObs.setDcid(dcid);
svObs.setPvs(PropertyValues.newBuilder().build());
return svObs.build();
Expand Down Expand Up @@ -114,23 +114,53 @@ public void testBuildOptimizedMcfGraph() {
"count_person",
"country/USA",
Arrays.asList(
createStatVarObs("2020", 32.0, "obs1"),
createStatVarObs("2021", 33.0, "obs2"))))
createStatVarObs("2020", "32.0", "obs1"),
createStatVarObs("2021", "33.0", "obs2"))))
.build();
McfOptimizedGraph expected2 =
McfOptimizedGraph.newBuilder()
.setSvObsSeries(
createMcfStatVarObsSeries(
"count_person",
"country/India",
List.of(createStatVarObs("2022", 36.0, "obs4"))))
List.of(createStatVarObs("2022", "36.0", "obs4"))))
.build();

PAssert.that(result).containsInAnyOrder(expected1, expected2);
PipelineResult.State state = p.run().waitUntilFinish();
Assert.assertEquals(PipelineResult.State.DONE, state);
}

@Test
public void testBuildOptimizedMcfGraph_preservesSigFigs() {
options.setStableUniqueNames(PipelineOptions.CheckEnabled.OFF);
p.getCoderRegistry()
.registerCoderForClass(
McfStatVarObsSeries.Key.class, ProtoCoder.of(McfStatVarObsSeries.Key.class));

PCollection<McfGraph> input =
p.apply(
Create.of(
createStatVarObservationGraph(
"obsSigFig", "measurement_rate", "country/USA", "2020", "12.5000")));

PCollection<McfOptimizedGraph> result =
PipelineUtils.buildOptimizedMcfGraph("testSigFig", input);

McfOptimizedGraph expected =
McfOptimizedGraph.newBuilder()
.setSvObsSeries(
createMcfStatVarObsSeries(
"measurement_rate",
"country/USA",
List.of(createStatVarObs("2020", "12.5000", "obsSigFig"))))
.build();

PAssert.that(result).containsInAnyOrder(expected);
PipelineResult.State state = p.run().waitUntilFinish();
Assert.assertEquals(PipelineResult.State.DONE, state);
}

@Test
public void testCombineGraphNodes() {
// Input Graph 1
Expand Down
4 changes: 2 additions & 2 deletions tool/src/test/resources/org/datacommons/tool/genmcf/manyinconsistent/output/report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@
"file": "acre.csv",
"lineNumber": "3"
},
"userMessage": "Found nodes with different values for the same StatVarObservation :: observationAbout: 'geoId/01', variableMeasured: 'Some_StatVar', observationDate: '2020-01-01', value1: 1.0, value2: 2.0",
"userMessage": "Found nodes with different values for the same StatVarObservation :: observationAbout: 'geoId/01', variableMeasured: 'Some_StatVar', observationDate: '2020-01-01', value1: 1, value2: 2",

@gmechali gmechali Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fyi the reason we are now removing the "1.0" and instead showing "1" isnt' about a loss of sigfigs but actually preservation of sigfigs.

We used to convert the 1 to a float making it 1.0. Now we preserve the value.

"counterKey": "Sanity_InconsistentSvObsValues"
}, {
"level": "LEVEL_ERROR",
"location": {
"file": "acre.csv",
"lineNumber": "4"
},
"userMessage": "Found nodes with different values for the same StatVarObservation :: observationAbout: 'geoId/01', variableMeasured: 'Some_StatVar', observationDate: '2020-01-01', value1: 1.0, value2: 3.0",
"userMessage": "Found nodes with different values for the same StatVarObservation :: observationAbout: 'geoId/01', variableMeasured: 'Some_StatVar', observationDate: '2020-01-01', value1: 1, value2: 3",
"counterKey": "Sanity_InconsistentSvObsValues"
}, {
"level": "LEVEL_ERROR",
Expand Down
2 changes: 1 addition & 1 deletion tool/src/test/resources/org/datacommons/tool/genmcf/statchecks/output/report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@
"file": "covid.csv",
"lineNumber": "31"
},
"userMessage": "Found nodes with different values for the same StatVarObservation :: observationAbout: 'geoId/06', variableMeasured: 'CumulativeCount_MedicalTest_ConditionCOVID_19_Positive', observationDate: '2020-02-03', value1: 3.0, value2: 8.0",
"userMessage": "Found nodes with different values for the same StatVarObservation :: observationAbout: 'geoId/06', variableMeasured: 'CumulativeCount_MedicalTest_ConditionCOVID_19_Positive', observationDate: '2020-02-03', value1: 3, value2: 8",
"counterKey": "Sanity_InconsistentSvObsValues"
}],
"statsCheckSummary": [{
Expand Down
Loading