Skip to content

Cloudsqlpostgresql #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ragini2 wants to merge 2 commits into
base: cloudsqlpostgresql
Choose a base branch
from
Open

Cloudsqlpostgresql #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b2514b1
CloudSqlPsql test scenarios
Ragini2 Dec 17, 2021
de0711a
cloudSqlPsql related test scenarios
Ragini2 Dec 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ jobs:
${{ runner.os }}-maven-${{ github.workflow }}
- name: Run tests
run: python3 e2e/src/main/scripts/run_e2e_test.py
env:
Cloud_Psql_User_Name: ${{secrets.Cloud_Psql_User_Name}}
Cloud_Psql_Password: ${{secrets.Cloud_Psql_Password}}
Cloud_Psql_ConnectionName: ${{secrets.Cloud_Psql_ConnectionName}}
- name: Upload report
uses: actions/upload-artifact@v2.2.4
if: always()
Expand Down
26 changes: 11 additions & 15 deletions src/e2e-test/features/Cloudsqlpostgresql_sink_designtime.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,33 @@
Feature: CloudSQLPostGreSQL Sink Design Time
Feature: CloudSQLPostGreSQL Sink and Error Validation

@CLDPGSQL @TC-Mandatory-fields
@cloudSQLPostgreSQL
Scenario Outline:Verify CloudSQLPostGreSQL Sink properties validation errors for mandatory fields
Given Open DataFusion Project to configure pipeline
When Target is CloudSQLPostGreSQL
Then Open CloudSQLPostGreSQL Properties
Then Open cloudSQLPostgreSQL Properties
Then Enter the CloudSQLPostGreSQL Sink Properties with blank property "<property>"
Then Validate mandatory property error for "<property>"
Examples:
| property |
| referenceName |
| database |
| property |
| referenceName |
| database |
| connectionName |
| tableName |
| tableName |
| jdbcPluginName |

@CLDPGSQL @TC-Invalid-TestData-for-DriverName_Field:
@cloudSQLPostgreSQL
Scenario: Verify Driver Name field validation error with invalid test data
Given Open DataFusion Project to configure pipeline
When Target is CloudSQLPostGreSQL
Then Open CloudSQLPostGreSQL Properties
Then Open cloudSQLPostgreSQL Properties
Then Enter Reference Name & Database Name with Test Data
Then Enter Table Name "pSqlTableNameCS" and Connection Name "pSqlConnectionNameValid"
Then Enter Table Name "cloudPsqlTableName" and Connection Name
Then Validate Connector properties
Then Enter Driver Name with Invalid value
Then Verify Driver Name field with Invalid value entered

@sawantpritam sawantpritam Dec 23, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify driver name field error for invalid value

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Then Close the CloudSQLPostGreSQL Properties

@CLDPGSQL @TC-Invalid-TestData-for-ReferenceName&ConnectionName
@cloudSQLPostgreSQL
Scenario: User is able to Open and enter invalid test data for Reference Name & Connection Name

@sawantpritam sawantpritam Dec 23, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are you validating in this scenarios ?

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed scenario as not valid

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed scenario name

Given Open DataFusion Project to configure pipeline
When Target is CloudSQLPostGreSQL
Expand All @@ -38,7 +38,3 @@ Feature: CloudSQLPostGreSQL Sink Design Time
Then Verify Connection Name with private instance type

@sawantpritam sawantpritam Dec 23, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you validating any error message ?

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added error validation

Then Close the CloudSQLPostGreSQL Properties





20 changes: 10 additions & 10 deletions src/e2e-test/features/Cloudsqlpostgresql_source_designtime.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
Feature: CloudSQLPostGreSQL Source Design Time
Feature: CloudSQLPostGreSQL Source Design Time and error validation

@CLDPGSQL @TC-Mandatory-fields
@cloudSQLPostgreSQL
Scenario Outline:Verify CloudSQLPostGreSQL Source properties validation errors for mandatory fields
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
Then Open CloudSQLPostGreSQL Properties
Then Open cloudSQLPostgreSQL Properties
Then Enter the CloudSQLPostGreSQL Source Properties with blank property "<property>"
Then Validate mandatory property error for "<property>"
Examples:
Expand All @@ -15,23 +15,23 @@ Feature: CloudSQLPostGreSQL Source Design Time
| importQuery |

@sawantpritam sawantpritam Dec 23, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alignment of | is not proper

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

| jdbcPluginName |

@CLDPGSQL @TC-Invalid-TestData-for-DriverName_Field:
Scenario: TC-CLDMYSQL-DSGN-03:Verify Driver Name field validation error with invalid test data
@cloudSQLPostgreSQL
Scenario:Verify Driver Name field validation error with invalid test data
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
Then Open CloudSQLPostGreSQL Properties
Then Open cloudSQLPostgreSQL Properties
Then Enter Reference Name & Database Name with Test Data
Then Enter Connection Name and Import Query "pSqlImportQuery"
Then Enter Connection Name and Import Query "cloudPsqlImportQuery"
Then Validate Connector properties
Then Enter Driver Name with Invalid value
Then Verify Driver Name field with Invalid value entered

@sawantpritam sawantpritam Dec 23, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as Sink validation

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Then Close the CloudSQLPostGreSQL Properties

@CLDPGSQL @TC-Invalid-TestData-for-ReferenceName&ConnectionName
Scenario: TC-CLDMYSQL-DSGN-04:Verify Reference Name & Connection Name field validation errors with invalid test data
@cloudSQLPostgreSQL
Scenario:Verify Reference Name & Connection Name field validation errors with invalid test data
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
Then Enter Reference Name & Connection Name with Invalid Test Data and import query "pSqlImportQuery"
Then Enter Reference Name & Connection Name with Invalid Test Data and import query "cloudPsqlImportQuery"
Then Verify Reference Name Connection Name Fields with Invalid Test Data

@sawantpritam sawantpritam Dec 23, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verify error

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Then Enter Connection Name with private instance type
Then Verify Connection Name with private instance type

@sawantpritam sawantpritam Dec 23, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verify error

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Expand Down
264 changes: 264 additions & 0 deletions src/e2e-test/features/cloudSqlPsqlEndtoEnd.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,264 @@
Feature: End to End records transfer from cloudSQLPostgreSQL to BigQuery and GCS

@cloudSQLPostgreSQL-e2e
Scenario:Verify all the records transfer from cloudSQLPostgreSQL to Bigquery supporting different data types
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Target is BigQuery
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using query "cloudPSQLDBImportQueryForAll"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Open BigQuery Target Properties
Then Enter the BigQuery Target Properties for table "cloudPsqlBigQuery"
Then Validate Bigquery properties
Then Close the BigQuery properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "BigQuery" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null

@sawantpritam sawantpritam Dec 23, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Schema validation needs to be changed

@Ragini2 Ragini2 Dec 23, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pending to change column

Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Get Count of no of records transferred to BigQuery in "cloudPsqlBigQuery"
Then Validate records out from cloudSQLPostgreSQL is equal to records transferred in BigQuery "cloudPsqlBigQuery" output records

@cloudSQLPostgreSQL-e2e
Scenario:Verify all the duplicate records are fetched and transferred to BigQuery
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Target is BigQuery
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using query "cloudPSQLDBImportQueryDuplicate" for duplicate values "cloudPSQLSplitColumnDuplicateValue"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Open BigQuery Target Properties
Then Enter the BigQuery Target Properties for table "cloudPsqlBigQuery"
Then Validate Bigquery properties
Then Close the BigQuery properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "BigQuery" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null
Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Get Count of no of records transferred to BigQuery in "cloudPsqlBigQuery"
Then Validate records out from cloudSQLPostgreSQL is equal to records transferred in BigQuery "cloudPsqlBigQuery" output records

@cloudSQLPostgreSQL-e2e
Scenario Outline: Verify records get transferred on combining different tables using joins
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Target is BigQuery
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using different join queries "<cloudPSQLDBImportQueryJoins>"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Open BigQuery Target Properties
Then Enter the BigQuery Target Properties for table "cloudPsqlBigQuery"
Then Validate Bigquery properties
Then Close the BigQuery properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "BigQuery" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null
Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Get Count of no of records transferred to BigQuery in "cloudPsqlBigQuery"
Then Validate records out from cloudSQLPostgreSQL is equal to records transferred in BigQuery "cloudPsqlBigQuery" output records
Examples:
| cloudPSQLDBImportQueryJoins |
| cloudPSQLDBImportQueryInnerJoin |
| cloudPSQLDBImportQueryLeftJoin |
| cloudPSQLDBImportQueryRightJoin |
| cloudPSQLDBImportQueryOuterJoin |

@cloudSQLPostgreSQL-e2e
Scenario:Verify only distinct records are transferred
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Target is BigQuery
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using query "cloudPSQLDBImportQueryDistinct" for distinct values "cloudPSQLSplitColumnDistinctValue"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Open BigQuery Target Properties
Then Enter the BigQuery Target Properties for table "cloudPsqlBigQuery"
Then Validate Bigquery properties
Then Close the BigQuery properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "BigQuery" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null
Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Get Count of no of records transferred to BigQuery in "cloudPsqlBigQuery"
Then Validate records out from cloudSQLPostgreSQL is equal to records transferred in BigQuery "cloudPsqlBigQuery" output records

@cloudSQLPostgreSQL-e2e
Scenario:Verify records with maximum values are transferred from cloudPSQL to BigQuery
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Target is BigQuery
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using query "cloudPSQLDBImportQueryForMax" for max values "cloudPSQLSplitColumnMaxValue"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Open BigQuery Target Properties
Then Enter the BigQuery Target Properties for table "cloudPsqlBigQuery"
Then Validate Bigquery properties
Then Close the BigQuery properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "BigQuery" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null
Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Get Count of no of records transferred to BigQuery in "cloudPsqlBigQuery"
Then Validate records out from cloudSQLPostgreSQL is equal to records transferred in BigQuery "cloudPsqlBigQuery" output records

@cloudSQLPostgreSQL-e2e
Scenario:Verify records with minimum values are transferred from cloudPSQL to BigQuery
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Target is BigQuery
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using query "cloudPSQLDBImportQueryForMin" for min values "cloudPSQLSplitColumnMinValue"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Open BigQuery Target Properties
Then Enter the BigQuery Target Properties for table "cloudPsqlBigQuery"
Then Validate Bigquery properties
Then Close the BigQuery properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "BigQuery" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null
Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Get Count of no of records transferred to BigQuery in "cloudPsqlBigQuery"
Then Validate records out from cloudSQLPostgreSQL is equal to records transferred in BigQuery "cloudPsqlBigQuery" output records


@cloudSQLPostgreSQL-e2e
Scenario Outline: Verify all the records transfer from cloudSQLPostgreSQL to Bigquery for different where clause
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Target is BigQuery
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using query "<cloudPostgresSQLDatabaseImportQuery>" for max and min "<cloudPostgresSQLSplitColumnValues>"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Open BigQuery Target Properties
Then Enter the BigQuery Target Properties for table "cloudPsqlBigQuery"
Then Validate Bigquery properties
Then Close the BigQuery properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "BigQuery" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null
Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Get Count of no of records transferred to BigQuery in "cloudPsqlBigQuery"
Then Validate records out from cloudSQLPostgreSQL is equal to records transferred in BigQuery "cloudPsqlBigQuery" output records
Examples:
| cloudPostgresSQLDatabaseImportQuery | cloudPostgresSQLSplitColumnValues |
| cloudPSQLDBImportQueryForBetween | cloudPSQLSplitColumnBetweenValue |
| cloudPSQLDBImportQueryForIn | cloudPSQLSplitColumnInValue |
| cloudPSQLDBImportQueryNotIn | cloudPSQLSplitColumnNotInValue |
| cloudPSQLDBImportQueryOrderBy | cloudPSQLSplitColumnOrderByValue |


@cloudSQLPostgreSQL-e2e
Scenario:Verify records are transferred from cloudSQLPostgreSQL to BigQuery using Bounding Query
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Target is BigQuery
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using query "cloudPSQLQuery" for max values "cloudPSQLSplitColumnMaxValue" with bounding query "cloudPSQLDBBoundingQuery" and "cloudPsqlNoOfSplits"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Open BigQuery Target Properties
Then Enter the BigQuery Target Properties for table "cloudPsqlBigQuery"
Then Validate Bigquery properties
Then Close the BigQuery properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "BigQuery" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null
Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Get Count of no of records transferred to BigQuery in "cloudPsqlBigQuery"
Then Validate records out from cloudSQLPostgreSQL is equal to records transferred in BigQuery "cloudPsqlBigQuery" output records

@cloudSQLPostgreSQL-e2e
Scenario:Verify all the records transfer from cloudSQLPostgreSQL to GCS supporting different data types
Given Open DataFusion Project to configure pipeline
When Source is CloudSQLPostGreSQL
When Sink is GCS
Then Open cloudSQLPostgreSQL Properties
Then Enter the cloudSQLPostgreSQL properties for database "cloudPsqlDbName" using query "cloudPSQLDBImportQueryForAll"
Then Capture output schema
Then Validate cloudSQLPostgreSQL properties
Then Close the cloudSQLPostgreSQL properties
Then Enter the GCS Properties
Then Close the GCS Properties
Then Connect Source as "CloudSQL-PostgreSQL" and sink as "GCS" to establish connection
Then Add pipeline name
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Click on PreviewData for cloudSQLPostgreSQL
Then Verify Preview output schema is not null
Then Close the Preview and deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Then Open the Logs and capture raw logs
Then Validate the output record count
Loading