nonprojects / tgbox Goto Github PK

View Code? Open in Web Editor NEW

31.0 2.0 4.0 2.8 MB

Encrypted cloud storage Protocol based on Telegram

Home Page: https://tgbox.readthedocs.io/

License: GNU Lesser General Public License v2.1

Python 100.00%

telegram cloud-storage async api encrypted-files python3

tgbox's People

Contributors

Stargazers

Watchers

Forkers

qiangweihewu bellyfat nonoriginalnamereally master-in-science

tgbox's Issues

Download from specified offset

A way to download partially downloaded DecryptedRemoteBoxFile

Bug with retrieving thumbnail from Telegram file

See LonamiWebs/Telethon#3664.

tgbox/tgbox/api.py

Lines 2200 to 2209 in f121bed

    
           async def get_preview(self, quality: int=1) -> bytes: 
        
               if not self._client_initialized: 
        
                   self.ta = await self.ta 
        
                   self._client_initialized = True 
        
               return await self.ta.TelegramClient.download_media( 
        
                   message = self.doc_pic,  
        
                   thumb = quality, 
        
                   file = bytes 
        
               )

Try to multiprocess fastelethon.py (upload/download)

Because of GIL Python always runs on one core, so event loop too. We can create CPU_COUNT processes and upload files in parallel, but there is two big problems: the first, most obvious, is possible FloodWaitError on Telegram side, the second is pickle, it just doesn't work with coroutines and big classes.

aiomultiprocess.Pool doesn't work, as it rely on pickle. dill instead of pickle.... doesn't work. We need to fully rewrite fastelethon, so it's coros will return parts from which we can reconstruct objects in other Process.

It looks good in theory, but FloodWaitError and GIL... meh, i don't want to do it now. If anyone has ideas - feel free to publish them here.

Independence from Telegram

We need to implement a bunch of things:

Test with PyAES

ReadTheDocs doesn't catch modules

Maybe because we removed a requirements.txt file.
May be fix: https://stackoverflow.com/questions/57835222/readthedocs-config-for-setup-py-installation

Better file previews

LimitInvalidError on big Filedata

The Telethon library throws LimitInvalidError when we try to download a RemoteBoxFile with big filedata (e.g where folder len equals to FOLDERNAME_MAX)

tgbox/api.py

Lines 929 to 931 in 17025f3

    
           filedata_len = bytes_to_int(dec_navbytes[:3],signed=False) 
        
           preview_len = bytes_to_int(dec_navbytes[3:],signed=False) 
        
           #TODO: BIG FILEDATA LEN IS BROKEN

This error already was reported to Telethon devs.
LonamiWebs/Telethon#3199 (comment)

Anyway, we should investigate this.

Custom attributes on RemoteBoxFile

Outdated. See #4 (comment)

We have a 255-byte field called comment. I suggest to use it for determining file types.

My thoughts about this:

;msg; determines that this file is Text-message;
;rep;5; determines that this file is Reply to another file with ID=5;
;1;alb;{alb_id}; determines that this file is part of Album. 1 here is order. alb_id is unique Album ID. Maybe 4 random bytes?
;pubkey;sgn;signature; determines that this file was signed by pubkey.
...

Windows doesn't close SQL DB file, *LocalBox.delete()

Add offset to the fastelethon.py upload/download functions

Currently we download all bytes before offset. This isn't a big problem, cuz typically we want to skip RemoteBoxFileMetadata (max 2 MB, typically 200B~3KB) and download file, so we can do this in future.

Bug on iterating files over RemoteBox

When RemoteBox has imported (forwarded) file, and DecryptedLocalBox hasn't FileKey for this file, iterating over RemoteBox.files(), manifests a lag for ~5 seconds or more.

repr all classes

To __repr__ or not to __repr__?

PyTest

Server closed the connection: 0 bytes read on a total of 8 expected bytes

:2023-05-04 ^ 21:45:56: WARNING:telethon.network.connection.connection~_recv_loop{334} ::: Server closed the connection: 0 bytes read on a total of 8 expected bytes
:2023-05-04 ^ 21:45:58: ERROR:telethon.network.mtprotosender~_recv_loop{519} ::: Unhandled error while receiving data
Traceback (most recent call last):
  File "/home/pronet/.local/lib/python3.9/site-packages/telethon/network/mtprotosender.py", line 505, in _recv_loop
    body = await self._connection.recv()
  File "/home/pronet/.local/lib/python3.9/site-packages/telethon/network/connection/connection.py", line 301, in recv
    raise err
  File "/home/pronet/.local/lib/python3.9/site-packages/telethon/network/connection/connection.py", line 332, in _recv_loop
    data = await self._recv()
  File "/home/pronet/.local/lib/python3.9/site-packages/telethon/network/connection/connection.py", line 369, in _recv
    return await self._codec.read_packet(self._reader)
  File "/home/pronet/.local/lib/python3.9/site-packages/telethon/network/connection/tcpfull.py", line 25, in read_packet
    packet_len_seq = await reader.readexactly(8)  # 4 and 4
  File "/usr/lib/python3.9/asyncio/streams.py", line 721, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 8 expected bytes

result = await gather(
    rb.push_file(await dlb.make_file(open('test.jpg','rb'))),
    rb.push_file(await dlb.make_file(open('test.jpg','rb'))),
    rb.push_file(await dlb.make_file(open('test.jpg','rb')))
)
print(result)

Traceback (most recent call last):
  File "/home/non/Scripts/py_projects/tgbox-env/test.py", line 202, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/non/Scripts/py_projects/tgbox-env/test.py", line 77, in main
    result = await gather(
  File "/home/non/Scripts/py_projects/tgbox-env/tgbox/api.py", line 415, in push_file
    await ff.make_local(file_message.id, 
  File "/home/non/Scripts/py_projects/tgbox-env/tgbox/api.py", line 1992, in make_local
    maybe_file = await self.dlb._tgbox_db.Files.select_once(
  File "/home/non/Scripts/py_projects/tgbox-env/tgbox/db.py", line 44, in select_once
    return await (self.select(sql_tuple=sql_tuple)).__anext__()
StopAsyncIteration

	async def get_preview(self, quality: int=1) -> bytes:
	if not self._client_initialized:
	self.ta = await self.ta
	self._client_initialized = True

	return await self.ta.TelegramClient.download_media(
	message = self.doc_pic,
	thumb = quality,
	file = bytes
	)

	filedata_len = bytes_to_int(dec_navbytes[:3],signed=False)
	preview_len = bytes_to_int(dec_navbytes[3:],signed=False)
	#TODO: BIG FILEDATA LEN IS BROKEN