qlixed / python-memwiper Goto Github PK
View Code? Open in Web Editor NEWNow you can overwrite with garbage your inmutable strings without issues
License: MIT License
python-memwiper's People
Contributors
Stargazers
Watchers
python-memwiper's Issues
[Win] Windows fatal exception: access violation in test_wipe
During the windows test we see the following error:
tests/test_memwiper.py::test_wipe[Hola ni\xf1o!\n] Windows fatal exception: access violation
Test for windows is failing, always. The full log of the last run is:
[5080] C:\Users\travis\build\qlixed\python-memwiper$ 'C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\Scripts\python.EXE' -m pip install --exists-action w '.tox\.tmp\package\1\memwiper-0.9.0.zip' >.tox\py38-nocov\log\py38-nocov-2.log
[3656] C:\Users\travis\build\qlixed\python-memwiper$ 'C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\Scripts\python.EXE' -m pip freeze >.tox\py38-nocov\log\py38-nocov-3.log
py38-nocov installed: atomicwrites==1.3.0,attrs==19.3.0,colorama==0.4.3,memwiper==0.9.0,more-itertools==8.0.2,packaging==19.2,pluggy==0.13.1,py==1.8.1,pyparsing==2.4.6,pytest==5.3.2,pytest-travis-fold==1.3.0,six==1.13.0,wcwidth==0.1.8
py38-nocov run-test-pre: PYTHONHASHSEED='681'
py38-nocov run-test: commands[0] | python setup.py clean --all build_ext --force --inplace
[4804] C:\Users\travis\build\qlixed\python-memwiper$ 'C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\Scripts\python.EXE' setup.py clean --all build_ext --force --inplace
running clean
'build\lib.win-amd64-3.8' does not exist -- can't clean it
'build\bdist.win-amd64' does not exist -- can't clean it
'build\scripts-3.8' does not exist -- can't clean it
running build_ext
building 'memwiper.c_src.memwiper_core' extension
creating build
creating build\temp.win-amd64-3.8
creating build\temp.win-amd64-3.8\Release
creating build\temp.win-amd64-3.8\Release\src
creating build\temp.win-amd64-3.8\Release\src\memwiper
creating build\temp.win-amd64-3.8\Release\src\memwiper\c_src
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Isrc\memwiper\c_src -Ic:\python38\include -Ic:\python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcsrc\memwiper\c_src\memwiper_core.c /Fobuild\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_core.obj
memwiper_core.c
src\memwiper\c_src\memwiper_core.c(40): warning C4244: '=': conversion from 'Py_ssize_t' to 'int', possible loss of data
src\memwiper\c_src\memwiper_core.c(50): warning C4566: character represented by universal-character-name '\uFFFF' cannot be represented in the current code page (1252)
src\memwiper\c_src\memwiper_core.c(50): warning C4566: character represented by universal-character-name '\U000FFFFF' cannot be represented in the current code page (1252)
src\memwiper\c_src\memwiper_core.c(90): warning C4244: '=': conversion from 'Py_ssize_t' to 'int', possible loss of data
creating C:\Users\travis\build\qlixed\python-memwiper\build\lib.win-amd64-3.8
creating C:\Users\travis\build\qlixed\python-memwiper\build\lib.win-amd64-3.8\memwiper
creating C:\Users\travis\build\qlixed\python-memwiper\build\lib.win-amd64-3.8\memwiper\c_src
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\python38\Libs /LIBPATH:C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\libs /LIBPATH:C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_memwiper_core build\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_core.obj /OUT:build\lib.win-amd64-3.8\memwiper\c_src\memwiper_core.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_core.cp38-win_amd64.lib
Creating library build\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_core.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_core.cp38-win_amd64.exp
Generating code
Finished generating code
building 'memwiper.c_src.memwiper_utils' extension
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Isrc\memwiper\c_src -Ic:\python38\include -Ic:\python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcsrc\memwiper\c_src\memwiper_utils.c /Fobuild\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_utils.obj
memwiper_utils.c
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\python38\Libs /LIBPATH:C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\libs /LIBPATH:C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_memwiper_utils build\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_utils.obj /OUT:build\lib.win-amd64-3.8\memwiper\c_src\memwiper_utils.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_utils.cp38-win_amd64.lib
Creating library build\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_utils.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\src\memwiper\c_src\memwiper_utils.cp38-win_amd64.exp
Generating code
Finished generating code
copying build\lib.win-amd64-3.8\memwiper\c_src\memwiper_core.cp38-win_amd64.pyd -> src\memwiper\c_src
copying build\lib.win-amd64-3.8\memwiper\c_src\memwiper_utils.cp38-win_amd64.pyd -> src\memwiper\c_src
py38-nocov run-test: commands[1] | pytest -vv --ignore=src
[1016] C:\Users\travis\build\qlixed\python-memwiper$ 'C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\Scripts\pytest.EXE' -vv --ignore=src
============================= test session starts =============================
platform win32 -- Python 3.8.0, pytest-5.3.2, py-1.8.1, pluggy-0.13.1 -- c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\scripts\python.exe
cachedir: .tox\py38-nocov\.pytest_cache
rootdir: C:\Users\travis\build\qlixed\python-memwiper, inifile: tox.ini, testpaths: tests
plugins: travis-fold-1.3.0
collecting ... collected 83 items
tests/test_memwiper.py::test_wipe[\u0645\u0631\u062d\u0628\u0627 \u0637\u0641\u0644!\n] PASSED [ 1%]
tests/test_memwiper.py::test_wipe[\u1787\u17c6\u179a\u17b6\u1794\u179f\u17bd\u179a\u1780\u17bc\u1793!\n] PASSED [ 2%]
tests/test_memwiper.py::test_wipe[Dobr\xfd den d\xedt\u011b!\n] PASSED [ 3%]
tests/test_memwiper.py::test_wipe[\u4f60\u597d\u5b69\u5b50\uff01\n] PASSED [ 4%]
tests/test_memwiper.py::test_wipe[\uc548\ub155 \uc598\uc57c!\n] PASSED [ 6%]
tests/test_memwiper.py::test_wipe[Dobr\xfd de\u0148 die\u0165a!\n] PASSED [ 7%]
tests/test_memwiper.py::test_wipe[Hola ni\xf1o!\n] Windows fatal exception: access violation
Current thread 0x000012e4 (most recent call first):
File "C:\Users\travis\build\qlixed\python-memwiper\tests\test_memwiper.py", line 21 in test_wipe
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\python.py", line 166 in pytest_pyfunc_call
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\callers.py", line 187 in _multicall
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\hooks.py", line 286 in __call__
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\python.py", line 1435 in runtest
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\runner.py", line 131 in pytest_runtest_call
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\callers.py", line 187 in _multicall
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\hooks.py", line 286 in __call__
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\runner.py", line 207 in <lambda>
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\runner.py", line 234 in from_call
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\runner.py", line 206 in call_runtest_hook
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\runner.py", line 182 in call_and_report
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\runner.py", line 96 in runtestprotocol
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\runner.py", line 81 in pytest_runtest_protocol
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\callers.py", line 187 in _multicall
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\hooks.py", line 286 in __call__
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\main.py", line 270 in pytest_runtestloop
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\callers.py", line 187 in _multicall
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\hooks.py", line 286 in __call__
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\main.py", line 246 in _main
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\main.py", line 196 in wrap_session
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\main.py", line 239 in pytest_cmdline_main
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\callers.py", line 187 in _multicall
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\pluggy\hooks.py", line 286 in __call__
File "c:\users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\lib\site-packages\_pytest\config\__init__.py", line 91 in main
File "C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\Scripts\pytest.EXE\__main__.py", line 7 in <module>
File "c:\python38\Lib\runpy.py", line 85 in _run_code
File "c:\python38\Lib\runpy.py", line 192 in _run_module_as_main
ERROR: InvocationError for command 'C:\Users\travis\build\qlixed\python-memwiper\.tox\py38-nocov\Scripts\pytest.EXE' -vv --ignore=src (exited with code 3221225477)
Build Failed for a tox coverage execution error
Need to add '-' in front of the command in the line:
python-memwiper/ci/templates/tox.ini
Line 104 in 595d207
As is shown on:
https://tox.readthedocs.io/en/latest/example/basic.html#ignoring-exit-code
Update the CI scripts
Cookie cutter used for this projects updated the management of the ci and tox environments, this require and update.
Add some performance test
Make a test that get some performance numbers to compare over the platforms and python releases.
Basic Test on Mac Platform
So far the code have only Python functions. This was indeed a rational decision to make the code portable without any additional deps.
We need to make sure that this works on Mac OS.
Expand the test suite
Make more tests.
Evaluate the ctypes.pythonapi usage.
Checkout the ctypes.pythonapi to see if we can make the same code with that, converting the C code and making the overall project more compatible and stable.
Error about Interned strings
On interpreter close we get an error trying to remove the interned string, as the overwrited one is not in the interned dict.
Checks for mem leaks
Make some test that let us verify if there is some memory leakage.
So far I will use valgrind and pympler. But just if both are installed.
This initially will be oriented for linux.
Basic Test on Windows Platform
So far the code have only Python functions. This was indeed a rational decision to make the code portable without any additional deps.
We need to make sure that this works on windows.
Make a CI pipeline
Add CI!
Overwrite behavior
Currently I simply overwrite the buffer and nothing else. This is done in this way to avoid memory leaks and other kind of issues that might appear if i change the object buffer size, this leads us to get the '\0x0\0x0...' string.
With this in mind, here is some ideas about the current situation:
- Make a overwrite-and-get-new-string: So we get a correct "empty" string: "". This might add an extra decrement of the reference of the object.
- Let the '\0x0...' format as a kind-of-debug option, to double check the data
Modularize Code
Make some cleanup of the current code, all the functions that don't have a direct relation with the wipe, needs to go to some kind of 'utils' or 'debug' submodule.
Change CI scripts to use only travis
Travis now support windows, so I prefer to move over it, deprecating appveyor.
Make coretest.py concept script for Mac
Make a coretest.py compatible for Mac.
If this is not possible then make a similar concept script.
Make coretest.py concept script for Windows
Make a coretest.py compatible for Windows.
If this is not possible then make a similar concept script.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.