Comments (5)
we just got an update: for natively specified file paths to local filesysten ie C:\my\path
if the path does not exist there's no error message. if the path exists the pipeline does not see any files. so certainly something is wrong here
from dlt.
@rudolfix, okay, I think I reproduced it:
Figuring out what's wrong there.
from dlt.
@rudolfix, I wonder what's the meaning of this line:
dlt/dlt/common/storages/fsspec_filesystem.py
Line 287 in 20664b2
from dlt.
@IlyaFaer this strips leading "//" from the front but IDK why.
I think this part:
# make that absolute path on a file://
if bucket_url_parsed.scheme == "file" and not file.startswith("/"):
file = f"/{file}"
file_name = posixpath.relpath(file, bucket_url_no_schema)
file_url = bucket_url_parsed._replace(
path=posixpath.join(bucket_url_parsed.path, file_name)
).geturl()
destroys some of the local paths
you'll need to debug that more
from dlt.
@rudolfix, yes, I brought a couple of fixes into your code, but I see that fsspect
package code is also breaking the path in some way. Checking what's wrong there yet...
from dlt.
Related Issues (20)
- Error when using refresh="drop_resources" and change the write_disposition from append to merge
- file format inconsistencies with variant columns
- error when loading multiple resources into `delta` table format with multithreading HOT 1
- Support for DuckDB 1.1.0 HOT 1
- Docs: replace absolute links with relative links
- Add sftp configuration to filesystem source docs
- Feature: report missing "parent" in schema yaml HOT 2
- Feature: report field type differences between DB schema and schema yaml HOT 1
- rest_api: Allow specifying custom session HOT 1
- Default hint of "unique" does not add a unique index
- Empty Response (e.g. HTTP 204) causes ignore response_actions to fail HOT 2
- Can't use AzureServicePrincipalCredentials with Databricks destination and Azure staging
- Links in Tip boxes while in dark mode are hard/impossible to read HOT 1
- DB2 sql_database source | dlt queries wrong system tables for iseries HOT 5
- Followup Work for dataset dataacess
- Docs: rest_api: document session param in client config
- When ingesting a sql_database with a UUID column, getting a flood of warnings
- rest_api: Suggest renaming `resolve` to `reference` in dependent resources
- Error loading data from MSSQL when using a compound merge key
- Unable to do an initial load into "bigquery" with any other write_disposition then "append"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dlt.