Coder Social home page Coder Social logo

go-w3s-client's Introduction

⚠️ DEPRECATED This repo will be archived on January 9, 2024 as this web3.storage API will no longer take new uploads. Please use the new client and API for future usage of web3.storage. Documentation for the new client can be found here. You can learn more about these changes here.

go-w3s-client

A client to the Web3.Storage API.

Demo: https://youtu.be/FLsQZ_ogeOg

Install

go get github.com/web3-storage/go-w3s-client

Usage

package main

import (
    "io/fs"
    "os"
    "github.com/web3-storage/go-w3s-client"
)

func main() {
    c, _ := w3s.NewClient(w3s.WithToken("<AUTH_TOKEN>"))
    f, _ := os.Open("images/pinpie.jpg")

    // OR add a whole directory:
    //
    //   f, _ := os.Open("images")
    //
    // OR create your own directory:
    //
    //   img0, _ := os.Open("aliens.jpg")
    //   img1, _ := os.Open("donotresist.jpg")
    //   f := w3fs.NewDir("images", []fs.File{img0, img1})

    // Write a file/directory
    cid, _ := c.Put(context.Background(), f)
    fmt.Printf("https://%v.ipfs.dweb.link\n", cid)

    // Retrieve a file/directory
    res, _ := c.Get(context.Background(), cid)
    
    // res is a http.Response with an extra method for reading IPFS UnixFS files!
    f, fsys, _ := res.Files()

    // List directory entries
    if d, ok := f.(fs.ReadDirFile); ok {
        ents, _ := d.ReadDir(0)
        for _, ent := range ents {
            fmt.Println(ent.Name())
        }
    }

    // Walk whole directory contents (including nested directories)
    fs.WalkDir(fsys, "/", func(path string, d fs.DirEntry, err error) error {
        info, _ := d.Info()
        fmt.Printf("%s (%d bytes)\n", path, info.Size())
        return err
    })

    // Open a file in a directory
    img, _ := fsys.Open("pinpie.jpg")
    // img.Stat()
    // img.Read(...)
    // img.Close()
}

See example for more.

API

pkg.go.dev Reference

Contribute

Feel free to dive in! Open an issue or submit PRs.

License

Dual-licensed under MIT + Apache 2.0

go-w3s-client's People

Contributors

3456091 avatar alanshaw avatar gipsh avatar iand avatar jorropo avatar yusefnapora avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-w3s-client's Issues

put with folder does not work

Hi,

The put function is not well enough tested.

I got this error while with the path 480p.

failed to upload segment: failed to upload segment folder: opening file 480p/480p_007.ts: open 480p/480p_007.ts: no such file or directory

image

It's strange because the file at 480p/480_...ts exist.

I am using the basic os.Open to open this folder.


Aside question :

  1. How to then retrieve all cid of each files uploaded ?
  2. Is it possible to hook somewhere to get the upload progress ?
  3. Is it possible to use the IPFS golang library instead of yours ?

Is go 1.18 not supported?

when I run the Usage with go 1.18 I got those errors

# github.com/web3-storage/go-w3s-client/fs/adapter
../../../.gvm/pkgsets/go1.18/global/pkg/mod/github.com/web3-storage/[email protected]/fs/adapter/unixfs.go:54:13: cannot use pathresolver.NewBasicResolver(unixFSFetcher) (value of type resolver.Resolver) as type *resolver.Resolver in struct literal:
	resolver.Resolver does not implement *resolver.Resolver (type *resolver.Resolver is pointer to interface, not interface)
../../../.gvm/pkgsets/go1.18/global/pkg/mod/github.com/web3-storage/[email protected]/fs/adapter/unixfs.go:69:32: fs.resolver.ResolveToLastNode undefined (type *resolver.Resolver is pointer to interface, not interface)

Client.Status can't always parse time values

I sporadically receive an error from Client.Status of "parsing time "" as "2006-01-02T15:04:05.999Z07:00": cannot parse "" as "2006") failed". Using the w3 client with the same CID shows the top-level created field contains a valid date, but the date fields in the deals array are null.

Build errors due to stale commit referenced in `go.mod`

The go.mod have:

github.com/ipfs/go-mfs v0.1.3-0.20210507195338-96fbfa122164

however this commit isn't in the github.com/ipfs/go-mfs repo anymore so trying to go build ./... yields:

go: github.com/ipfs/[email protected]: invalid version: unknown revision 96fbfa122164
go: downloading github.com/ipfs/go-mfs v0.1.3-0.20210507195338-96fbfa122164
go: github.com/ipfs/[email protected]: invalid version: unknown revision 96fbfa122164

Note: you / the CI might have it if :

  • You already downloaded this version and just reading it from your ~/go/pkg/mod.
  • You are not using GOPROXY="direct'" and just downloading it from proxy.golang.org.

Add X-Client header

Add X-Client http header so we can see in Sentry the client being used (for help debugging).

Frozen with large files

Upon uploading large files (~2GB) it taken up all of the RAM and swap then did freeze the Ubuntu machine it was running on. Disk read showed several times of the actual size of the file (~30GB). This happens when using the function putDirectory() with a folder containing only the aforementioned file.

go get error: package io/fs is not in GOROOT (/usr/lib/golang/src/io/fs)

when i run

go get github.com/web3-storage/go-w3s-client
/root/go/pkg/mod/github.com/web3-storage/[email protected]/fs/dir.go:5:2: package io/fs is not in GOROOT (/usr/lib/golang/src/io/fs)

my deploment env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn"
GOROOT="/usr/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/www/wwwroot/go2mall.liaoxx.top/mall-api/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build775891149=/tmp/go-build -gno-record-gcc-switches"

my server

 Static hostname: iZwz9cn0tgf34dj6phzffiZ
Transient hostname: ctl
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 20200218155604663518171520219378
           Boot ID: 8322b72124ca46cd922b0d09051462f3
    Virtualization: kvm
  Operating System: CentOS Linux 8
       CPE OS Name: cpe:/o:centos:centos:8
            Kernel: Linux 4.18.0-305.7.1.el8_4.x86_64
      Architecture: x86-64

my code:

package info

import (
	"context"
	"fmt"
	"github.com/web3-storage/go-w3s-client"
	"mime/multipart"
	"os"
)


func AdminUploadImageWeb3StorageBusiness(img *multipart.FileHeader, uid int64, f string, ext string) (r string, err error){
	var token ="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkaWQ6ZXRocjoweDhmMjRDNDNCMTUyRjBmNjU1OURFZmQzN0IxRTdkOWU3ZkJjN2IyMzYiLCJpc3MiOiJ3ZWIzLXN0b3JhZ2UiLCJpYXQiOjE2MzE1OTA0NjgzNjIsIm5hbWUiOiJsaW9uMTNicm8ifQ.gVJWN4d5nCD3Fse5pQuzPrEr8xOrFgxPDDV4gPWhykE"

	_Rs := ""
	src, err := img.Open()
	if err != nil {
		return _Rs, fmt.Errorf(fmt.Sprintf("%v", err))
	}
	defer src.Close()

	c, _ := w3s.NewClient(w3s.WithToken(token))
	filePnt, err := os.Open(img.Filename)
	cid, err := c.Put(context.Background(), filePnt)
	  fmt.Printf("https://%v.ipfs.dweb.link\n", cid)
	return
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.