Skip to content

Conversation

@grubberr
Copy link
Contributor

@grubberr grubberr commented Nov 30, 2022

Signed-off-by: Sergey Chvalyuk [email protected]

What

Try to fix https://github.com/airbytehq/oncall/issues/1115

pendulum.parse(, strict=False) - leaks memory

import pendulum

for _ in range(1000000):
    pendulum.parse("2021-04-08 14:23:30 +00:00", strict=False)

After 100000 calls RSS memory 20056 kBs
After 1000000 calls RSS memory 138328 kBs

How

Replace pendulum on dateutil

Recommended reading order

  1. x.java
  2. y.python

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <[email protected]>
@octavia-squidington-iv octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Nov 30, 2022
@grubberr
Copy link
Contributor Author

grubberr commented Dec 1, 2022

/test connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/3591087520
✅ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/3591087520
Python tests coverage:

Name                                  Stmts   Miss  Cover
---------------------------------------------------------
source_iterable/utils.py                  5      0   100%
source_iterable/__init__.py               2      0   100%
source_iterable/source.py                34      1    97%
source_iterable/slice_generators.py      71      5    93%
source_iterable/streams.py              352     38    89%
---------------------------------------------------------
TOTAL                                   464     44    91%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     199     93    53%   35, 41-43, 48, 53, 59, 65, 71, 77-79, 98, 103-105, 111-113, 119-120, 125-126, 131, 137, 146-155, 161-166, 181, 205, 236, 242, 250-255, 263-268, 276-289, 294-300, 307-318, 325-341
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1569    350    78%

Build Passed

Test summary info:

All Passed
@grubberr grubberr self-assigned this Dec 1, 2022
@grubberr
Copy link
Contributor Author

grubberr commented Dec 2, 2022

/publish connector=connectors/source-iterable

🕑 Publishing the following connectors:
connectors/source-iterable
https://github.com/airbytehq/airbyte/actions/runs/3600435144


Connector Did it publish? Were definitions generated?
connectors/source-iterable

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets December 2, 2022 10:07 Inactive
@grubberr grubberr merged commit 3c6f979 into master Dec 2, 2022
@grubberr grubberr deleted the grubberr/oncall-1115-source-iterable branch December 2, 2022 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/iterable

5 participants