add support for serving Google Cloud Storage over SFTP/SCP

Each user can be mapped with a Google Cloud Storage bucket or a bucket
virtual folder

README.md

@@ -22,7 +22,7 @@ Full featured and highly configurable SFTP server
 - Atomic uploads are configurable.
 - Support for Git repositories over SSH.
 - SCP and rsync are supported.
-- Support for serving S3 Compatible Object Storage over SFTP.
+- Support for serving local filesystem, S3 Compatible Object Storage and Google Cloud Storage over SFTP/SCP.
 - Prometheus metrics are exposed.
 - REST API for users management, backup, restore and real time reports of the active connections with possibility of forcibly closing a connection.
 - Web based interface to easily manage users and connections.
@@ -178,6 +178,7 @@ The `sftpgo` configuration file contains the following sections:
         - `http_notification_url`, a valid URL. Leave empty to disable.
     - `external_auth_program`, string. Absolute path to an external program to use for users authentication. See the "External Authentication" paragraph for more details.
     - `external_auth_scope`, integer. 0 means all supported authetication scopes (passwords, public keys and keyboard interactive). 1 means passwords only. 2 means public keys only. 4 means key keyboard interactive only. The flags can be combined, for example 6 means public keys and keyboard interactive
+    - `credentials_path`, string. It defines the directory for storing user provided credential files such as Google Cloud Storage credentials. This can be an absolute path or a path relative to the config dir
 - **"httpd"**, the configuration for the HTTP server used to serve REST API
     - `bind_port`, integer. The port used for serving HTTP requests. Set to 0 to disable HTTP server. Default: 8080
     - `bind_address`, string. Leave blank to listen on all available network interfaces. Default: "127.0.0.1"
@@ -232,7 +233,8 @@ Here is a full example showing the default config in JSON format:
       "http_notification_url": ""
     },
     "external_auth_program": "",
-    "external_auth_scope": 0
+    "external_auth_scope": 0,
+    "credentials_path": "credentials"
   },
   "httpd": {
     "bind_port": 8080,
@@ -270,10 +272,12 @@ Please note that to override configuration options with environment variables a
 
 ### Data provider initialization
 
-Before starting `sftpgo serve` a data provider must be configured.
+Before starting `sftpgo serve` please ensure that the configured dataprovider is properly initialized.
 
-SQL scripts to create the required database structure can be found inside the source tree [sql](./sql "sql") directory. The SQL scripts filename is, by convention, the date as `YYYYMMDD` and the suffix `.sql`. You need to apply all the SQL scripts for your database ordered by name, for example `20190828.sql` must be applied before `20191112.sql` and so on.  
-Example for `sqlite`: `find sql/sqlite/ -type f -iname '*.sql' -print |sort -n|xargs cat |sqlite3 sftpgo.db`
+SQL based data providers (SQLite, MySQL, PostgreSQL) requires the creation of a database containing the required tables. Memory and bolt data providers does not require an initialization.
+
+SQL scripts to create the required database structure can be found inside the source tree [sql](./sql "sql") directory. The SQL scripts filename is, by convention, the date as `YYYYMMDD` and the suffix `.sql`. You need to apply all the SQL scripts for your database ordered by name, for example `20190828.sql` must be applied before `20191112.sql` and so on.
+Example for `SQLite`: `find sql/sqlite/ -type f -iname '*.sql' -print | sort -n |xargs cat | sqlite3 sftpgo.db`
 
 ### Starting SFTGo in server mode
 
@@ -482,13 +486,13 @@ The HTTP request has a 15 seconds timeout.
 
 ## S3 Compabible Object Storage backends
 
-Each user can be mapped with an S3-Compatible bucket or a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP.
+Each user can be mapped to whole bucket or to a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP.
 
-Specifying a different `key_prefix` you can assign different virtual folders of the same bucket to different users. This is similar to a chroot directory for local filesystem. The virtual folder identified by `key_prefix` does not need to be pre-created.
+Specifying a different `key_prefix` you can assign different virtual folders of the same bucket to different users. This is similar to a chroot directory for local filesystem. Each SFTP/SCP user can only access to the assigned virtual folder and to its contents The virtual folder identified by `key_prefix` does not need to be pre-created.
 
 SFTPGo uses multipart uploads and parallel downloads for storing and retrieving files from S3.
 
-SFTPGo tries to automatically create the mapped bucket if it does not exists but it's a better idea to pre-create the bucket and to assign to it the wanted options such as automatic encryption and authorizations.
+The configured bucket must exist.
 
 Some SFTP commands doesn't work over S3:
 
@@ -504,6 +508,21 @@ Other notes:
 - For server side encryption you have to configure the mapped bucket to automatically encrypt objects.
 - A local home directory is still required to store temporary files.
 
+## Google Cloud Storage backend
+
+Each user can be mapped with a Google Cloud Storage bucket or a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP. This backend is very similar to the S3 backend and it has the same limitations.
+
+## Other Storage backends
+
+Adding new storage backends it's quite easy:
+
+- implement the [Fs interface](./vfs/vfs.go#L18 "interface for filesystem backends").
+- update the user method `GetFilesystem` to return the new backend
+- update the web interface and the REST API CLI
+- add the flags for the new storage backed to the `portable` mode
+
+Anyway some backends require a pay per use account (or they offer free account for a limited time period only), to be able to add support for such backends or to review pull requests please provide a test account. The test account must be available over the time to be able to maintain the backend and do basic tests before each new release.
+
 ## Portable mode
 
 SFTPGo allows to share a single directory on demand using the `portable` subcommand:
@@ -520,25 +539,29 @@ Usage:
   sftpgo portable [flags]
 
 Flags:
-  -C, --advertise-credentials     If the SFTP service is advertised via multicast DNS this flag allows to put username/password inside the advertised TXT record
-  -S, --advertise-service         Advertise SFTP service using multicast DNS (default true)
-  -d, --directory string          Path to the directory to serve. This can be an absolute path or a path relative to the current directory (default ".")
-  -f, --fs-provider int           0 means local filesystem, 1 S3 compatible
-  -h, --help                      help for portable
-  -l, --log-file-path string      Leave empty to disable logging
-  -p, --password string           Leave empty to use an auto generated value
-  -g, --permissions strings       User's permissions. "*" means any permission (default [list,download])
+  -C, --advertise-credentials         If the SFTP service is advertised via multicast DNS this flag allows to put username/password inside the advertised TXT record
+  -S, --advertise-service             Advertise SFTP service using multicast DNS (default true)
+  -d, --directory string              Path to the directory to serve. This can be an absolute path or a path relative to the current directory (default ".")
+  -f, --fs-provider int               0 means local filesystem, 1 Amazon S3 compatible, 2 Google Cloud Storage
+      --gcs-bucket string
+      --gcs-credentials-file string   Google Cloud Storage JSON credentials file
+      --gcs-key-prefix string         Allows to restrict access to the virtual folder identified by this prefix and its contents
+      --gcs-storage-class string
+  -h, --help                          help for portable
+  -l, --log-file-path string          Leave empty to disable logging
+  -p, --password string               Leave empty to use an auto generated value
+  -g, --permissions strings           User's permissions. "*" means any permission (default [list,download])
   -k, --public-key strings
       --s3-access-key string
       --s3-access-secret string
       --s3-bucket string
       --s3-endpoint string
-      --s3-key-prefix string      Allows to restrict access to the virtual folder identified by this prefix and its contents
+      --s3-key-prefix string          Allows to restrict access to the virtual folder identified by this prefix and its contents
       --s3-region string
       --s3-storage-class string
-  -s, --sftpd-port int            0 means a random non privileged port
-  -c, --ssh-commands strings      SSH commands to enable. "*" means any supported SSH command including scp (default [md5sum,sha1sum,cd,pwd])
-  -u, --username string           Leave empty to use an auto generated value
+  -s, --sftpd-port int                0 means a random non privileged port
+  -c, --ssh-commands strings          SSH commands to enable. "*" means any supported SSH command including scp (default [md5sum,sha1sum,cd,pwd])
+  -u, --username string               Leave empty to use an auto generated value
 ```
 
 In portable mode SFTPGo can advertise the SFTP service and, optionally, the credentials via multicast DNS, so there is a standard way to discover the service and to automatically connect to it.
@@ -592,6 +615,10 @@ For each account the following properties can be configured:
 - `s3_endpoint`, specifies s3 endpoint (server) different from AWS
 - `s3_storage_class`
 - `s3_key_prefix`, allows to restrict access to the virtual folder identified by this prefix and its contents
+- `gcs_bucket`, required for GCS filesystem
+- `gcs_credentials`, Google Cloud Storage JSON credentials base64 encoded
+- `gcs_storage_class`
+- `gcs_key_prefix`, allows to restrict access to the virtual folder identified by this prefix and its contents
 
 These properties are stored inside the data provider.
 

cmd/portable.go

@@ -1,6 +1,10 @@
 package cmd
 
 import (
+	"encoding/base64"
+	"fmt"
+	"io/ioutil"
+	"os"
 	"path/filepath"
 
 	"github.com/drakkan/sftpgo/dataprovider"
@@ -29,6 +33,10 @@ var (
 	portableS3Endpoint           string
 	portableS3StorageClass       string
 	portableS3KeyPrefix          string
+	portableGCSBucket            string
+	portableGCSCredentialsFile   string
+	portableGCSStorageClass      string
+	portableGCSKeyPrefix         string
 	portableCmd                  = &cobra.Command{
 		Use:   "portable",
 		Short: "Serve a single directory",
@@ -44,6 +52,24 @@ Please take a look at the usage below to customize the serving parameters`,
 			}
 			permissions := make(map[string][]string)
 			permissions["/"] = portablePermissions
+			portableGCSCredentials := ""
+			if portableFsProvider == 2 {
+				fi, err := os.Stat(portableGCSCredentialsFile)
+				if err != nil {
+					fmt.Printf("Invalid GCS credentials file: %v\n", err)
+					return
+				}
+				if fi.Size() > 1048576 {
+					fmt.Printf("Invalid GCS credentials file: %#v is too big %v/1048576 bytes\n", portableGCSCredentialsFile,
+						fi.Size())
+					return
+				}
+				creds, err := ioutil.ReadFile(portableGCSCredentialsFile)
+				if err != nil {
+					fmt.Printf("Unable to read credentials file: %v\n", err)
+				}
+				portableGCSCredentials = base64.StdEncoding.EncodeToString(creds)
+			}
 			service := service.Service{
 				ConfigDir:     defaultConfigDir,
 				ConfigFile:    defaultConfigName,
@@ -73,6 +99,12 @@ Please take a look at the usage below to customize the serving parameters`,
 							StorageClass: portableS3StorageClass,
 							KeyPrefix:    portableS3KeyPrefix,
 						},
+						GCSConfig: vfs.GCSFsConfig{
+							Bucket:       portableGCSBucket,
+							Credentials:  portableGCSCredentials,
+							StorageClass: portableGCSStorageClass,
+							KeyPrefix:    portableGCSKeyPrefix,
+						},
 					},
 				},
 			}
@@ -100,7 +132,8 @@ func init() {
 		"Advertise SFTP service using multicast DNS")
 	portableCmd.Flags().BoolVarP(&portableAdvertiseCredentials, "advertise-credentials", "C", false,
 		"If the SFTP service is advertised via multicast DNS this flag allows to put username/password inside the advertised TXT record")
-	portableCmd.Flags().IntVarP(&portableFsProvider, "fs-provider", "f", 0, "0 means local filesystem, 1 S3 compatible")
+	portableCmd.Flags().IntVarP(&portableFsProvider, "fs-provider", "f", 0, "0 means local filesystem, 1 Amazon S3 compatible, "+
+		"2 Google Cloud Storage")
 	portableCmd.Flags().StringVar(&portableS3Bucket, "s3-bucket", "", "")
 	portableCmd.Flags().StringVar(&portableS3Region, "s3-region", "", "")
 	portableCmd.Flags().StringVar(&portableS3AccessKey, "s3-access-key", "", "")
@@ -109,5 +142,10 @@ func init() {
 	portableCmd.Flags().StringVar(&portableS3StorageClass, "s3-storage-class", "", "")
 	portableCmd.Flags().StringVar(&portableS3KeyPrefix, "s3-key-prefix", "", "Allows to restrict access to the virtual folder "+
 		"identified by this prefix and its contents")
+	portableCmd.Flags().StringVar(&portableGCSBucket, "gcs-bucket", "", "")
+	portableCmd.Flags().StringVar(&portableGCSStorageClass, "gcs-storage-class", "", "")
+	portableCmd.Flags().StringVar(&portableGCSKeyPrefix, "gcs-key-prefix", "", "Allows to restrict access to the virtual folder "+
+		"identified by this prefix and its contents")
+	portableCmd.Flags().StringVar(&portableGCSCredentialsFile, "gcs-credentials-file", "", "Google Cloud Storage JSON credentials file")
 	rootCmd.AddCommand(portableCmd)
 }

config/config.go

@@ -84,6 +84,7 @@ func init() {
 			},
 			ExternalAuthProgram: "",
 			ExternalAuthScope:   0,
+			CredentialsPath:     "credentials",
 		},
 		HTTPDConfig: httpd.Conf{
 			BindPort:        8080,
@@ -178,6 +179,12 @@ func LoadConfig(configDir, configName string) error {
 		logger.Warn(logSender, "", "Configuration error: %v", err)
 		logger.WarnToConsole("Configuration error: %v", err)
 	}
+	if len(globalConf.ProviderConf.CredentialsPath) == 0 {
+		err = fmt.Errorf("invalid credentials path, reset to \"credentials\"")
+		globalConf.ProviderConf.CredentialsPath = "credentials"
+		logger.Warn(logSender, "", "Configuration error: %v", err)
+		logger.WarnToConsole("Configuration error: %v", err)
+	}
 	logger.Debug(logSender, "", "config file used: '%v', config loaded: %+v", viper.ConfigFileUsed(), getRedactedGlobalConf())
 	return err
 }

config/config_test.go

@@ -119,6 +119,27 @@ func TestInvalidExternalAuthScope(t *testing.T) {
 	os.Remove(configFilePath)
 }
 
+func TestInvalidCredentialsPath(t *testing.T) {
+	configDir := ".."
+	confName := tempConfigName + ".json"
+	configFilePath := filepath.Join(configDir, confName)
+	config.LoadConfig(configDir, "")
+	providerConf := config.GetProviderConf()
+	providerConf.CredentialsPath = ""
+	c := make(map[string]dataprovider.Config)
+	c["data_provider"] = providerConf
+	jsonConf, _ := json.Marshal(c)
+	err := ioutil.WriteFile(configFilePath, jsonConf, 0666)
+	if err != nil {
+		t.Errorf("error saving temporary configuration")
+	}
+	err = config.LoadConfig(configDir, tempConfigName)
+	if err == nil {
+		t.Errorf("Loading configuration with credentials path must fail")
+	}
+	os.Remove(configFilePath)
+}
+
 func TestSetGetConfig(t *testing.T) {
 	sftpdConf := config.GetSFTPDConfig()
 	sftpdConf.IdleTimeout = 3

dataprovider/bolt.go

@@ -319,6 +319,10 @@ func (p BoltProvider) dumpUsers() ([]User, error) {
 			if err != nil {
 				return err
 			}
+			err = addCredentialsToUser(&user)
+			if err != nil {
+				return err
+			}
 			users = append(users, user)
 		}
 		return err

dataprovider/dataprovider.go

@@ -16,6 +16,7 @@ import (
 	"errors"
 	"fmt"
 	"hash"
+	"io/ioutil"
 	"net"
 	"net/http"
 	"net/url"
@@ -85,6 +86,7 @@ var (
 	availabilityTicker     *time.Ticker
 	availabilityTickerDone chan bool
 	errWrongPassword       = errors.New("password does not match")
+	credentialsDirPath     string
 )
 
 // Actions to execute on user create, update, delete.
@@ -179,6 +181,10 @@ type Config struct {
 	// you can combine the scopes, for example 3 means password and public key, 5 password and keyboard
 	// interactive and so on
 	ExternalAuthScope int `json:"external_auth_scope" mapstructure:"external_auth_scope"`
+	// CredentialsPath defines the directory for storing user provided credential files such as
+	// Google Cloud Storage credentials. It can be a path relative to the config dir or an
+	// absolute path
+	CredentialsPath string `json:"credentials_path" mapstructure:"credentials_path"`
 }
 
 type keyboardAuthProgramResponse struct {
@@ -268,6 +274,9 @@ func Initialize(cnf Config, basePath string) error {
 			return err
 		}
 	}
+	if err := validateCredentialsDir(basePath); err != nil {
+		return err
+	}
 
 	if config.Driver == SQLiteDataProviderName {
 		err = initializeSQLiteProvider(basePath)
@@ -509,6 +518,25 @@ func validateFilters(user *User) error {
 	return nil
 }
 
+func saveGCSCredentials(user *User) error {
+	if user.FsConfig.Provider != 2 {
+		return nil
+	}
+	if len(user.FsConfig.GCSConfig.Credentials) == 0 {
+		return nil
+	}
+	decoded, err := base64.StdEncoding.DecodeString(user.FsConfig.GCSConfig.Credentials)
+	if err != nil {
+		return &ValidationError{err: fmt.Sprintf("could not validate GCS credentials: %v", err)}
+	}
+	err = ioutil.WriteFile(user.getGCSCredentialsFilePath(), decoded, 0600)
+	if err != nil {
+		return &ValidationError{err: fmt.Sprintf("could not save GCS credentials: %v", err)}
+	}
+	user.FsConfig.GCSConfig.Credentials = ""
+	return nil
+}
+
 func validateFilesystemConfig(user *User) error {
 	if user.FsConfig.Provider == 1 {
 		err := vfs.ValidateS3FsConfig(&user.FsConfig.S3Config)
@@ -524,9 +552,16 @@ func validateFilesystemConfig(user *User) error {
 			user.FsConfig.S3Config.AccessSecret = accessSecret
 		}
 		return nil
+	} else if user.FsConfig.Provider == 2 {
+		err := vfs.ValidateGCSFsConfig(&user.FsConfig.GCSConfig, user.getGCSCredentialsFilePath())
+		if err != nil {
+			return &ValidationError{err: fmt.Sprintf("could not validate GCS config: %v", err)}
+		}
+		return nil
 	}
 	user.FsConfig.Provider = 0
 	user.FsConfig.S3Config = vfs.S3FsConfig{}
+	user.FsConfig.GCSConfig = vfs.GCSFsConfig{}
 	return nil
 }
 
@@ -563,6 +598,9 @@ func validateUser(user *User) error {
 	if err := validateFilters(user); err != nil {
 		return err
 	}
+	if err := saveGCSCredentials(user); err != nil {
+		return err
+	}
 	return nil
 }
 
@@ -704,10 +742,24 @@ func HideUserSensitiveData(user *User) User {
 	user.Password = ""
 	if user.FsConfig.Provider == 1 {
 		user.FsConfig.S3Config.AccessSecret = utils.RemoveDecryptionKey(user.FsConfig.S3Config.AccessSecret)
+	} else if user.FsConfig.Provider == 2 {
+		user.FsConfig.GCSConfig.Credentials = ""
 	}
 	return *user
 }
 
+func addCredentialsToUser(user *User) error {
+	if user.FsConfig.Provider != 2 {
+		return nil
+	}
+	cred, err := ioutil.ReadFile(user.getGCSCredentialsFilePath())
+	if err != nil {
+		return err
+	}
+	user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString(cred)
+	return nil
+}
+
 func getSSLMode() string {
 	if config.Driver == PGSQLDataProviderName {
 		if config.SSLMode == 0 {
@@ -748,6 +800,25 @@ func startAvailabilityTimer() {
 	}()
 }
 
+func validateCredentialsDir(basePath string) error {
+	if filepath.IsAbs(config.CredentialsPath) {
+		credentialsDirPath = config.CredentialsPath
+	} else {
+		credentialsDirPath = filepath.Join(basePath, config.CredentialsPath)
+	}
+	fi, err := os.Stat(credentialsDirPath)
+	if err == nil {
+		if !fi.IsDir() {
+			return errors.New("Credential path is not a valid directory")
+		}
+		return nil
+	}
+	if !os.IsNotExist(err) {
+		return err
+	}
+	return os.MkdirAll(credentialsDirPath, 0700)
+}
+
 func checkDataprovider() {
 	err := provider.checkAvailability()
 	if err != nil {

dataprovider/memory.go

@@ -224,6 +224,10 @@ func (p MemoryProvider) dumpUsers() ([]User, error) {
 	}
 	for _, username := range p.dbHandle.usernames {
 		user := p.dbHandle.users[username]
+		err = addCredentialsToUser(&user)
+		if err != nil {
+			return users, err
+		}
 		users = append(users, user)
 	}
 	return users, err

dataprovider/sqlcommon.go

@@ -232,11 +232,14 @@ func sqlCommonDumpUsers(dbHandle *sql.DB) ([]User, error) {
 		defer rows.Close()
 		for rows.Next() {
 			u, err := getUserFromDbRow(nil, rows)
-			if err == nil {
-				users = append(users, u)
-			} else {
-				break
+			if err != nil {
+				return users, err
+			}
+			err = addCredentialsToUser(&u)
+			if err != nil {
+				return users, err
 			}
+			users = append(users, u)
 		}
 	}
 

dataprovider/user.go

@@ -55,9 +55,10 @@ type UserFilters struct {
 
 // Filesystem defines cloud storage filesystem details
 type Filesystem struct {
-	// 0 local filesystem, 1 Amazon S3 compatible
-	Provider int            `json:"provider"`
-	S3Config vfs.S3FsConfig `json:"s3config,omitempty"`
+	// 0 local filesystem, 1 Amazon S3 compatible, 2 Google Cloud Storage
+	Provider  int             `json:"provider"`
+	S3Config  vfs.S3FsConfig  `json:"s3config,omitempty"`
+	GCSConfig vfs.GCSFsConfig `json:"gcsconfig,omitempty"`
 }
 
 // User defines an SFTP user
@@ -73,7 +74,7 @@ type User struct {
 	ExpirationDate int64 `json:"expiration_date"`
 	// Password used for password authentication.
 	// For users created using SFTPGo REST API the password is be stored using argon2id hashing algo.
-	// Checking passwords stored with bcrypt, pbkdf2 and sha512crypt is supported too.
+	// Checking passwords stored with bcrypt, pbkdf2, md5crypt and sha512crypt is supported too.
 	Password string `json:"password,omitempty"`
 	// PublicKeys used for public key authentication. At least one between password and a public key is mandatory
 	PublicKeys []string `json:"public_keys,omitempty"`
@@ -113,6 +114,10 @@ type User struct {
 func (u *User) GetFilesystem(connectionID string) (vfs.Fs, error) {
 	if u.FsConfig.Provider == 1 {
 		return vfs.NewS3Fs(connectionID, u.GetHomeDir(), u.FsConfig.S3Config)
+	} else if u.FsConfig.Provider == 2 {
+		config := u.FsConfig.GCSConfig
+		config.CredentialFile = u.getGCSCredentialsFilePath()
+		return vfs.NewGCSFs(connectionID, u.GetHomeDir(), config)
 	}
 	return vfs.NewOsFs(connectionID, u.GetHomeDir()), nil
 }
@@ -321,7 +326,8 @@ func (u *User) GetBandwidthAsString() string {
 }
 
 // GetInfoString returns user's info as string.
-// Number of public keys, max sessions, uid and gid are returned
+// Storage provider, number of public keys, max sessions, uid,
+// gid, denied and allowed IP/Mask are returned
 func (u *User) GetInfoString() string {
 	var result string
 	if u.LastLogin > 0 {
@@ -329,7 +335,9 @@ func (u *User) GetInfoString() string {
 		result += fmt.Sprintf("Last login: %v ", t.Format("2006-01-02 15:04:05")) // YYYY-MM-DD HH:MM:SS
 	}
 	if u.FsConfig.Provider == 1 {
-		result += fmt.Sprintf("Storage: S3")
+		result += fmt.Sprintf("Storage: S3 ")
+	} else if u.FsConfig.Provider == 2 {
+		result += fmt.Sprintf("Storage: GCS ")
 	}
 	if len(u.PublicKeys) > 0 {
 		result += fmt.Sprintf("Public keys: %v ", len(u.PublicKeys))
@@ -410,6 +418,12 @@ func (u *User) getACopy() User {
 			StorageClass: u.FsConfig.S3Config.StorageClass,
 			KeyPrefix:    u.FsConfig.S3Config.KeyPrefix,
 		},
+		GCSConfig: vfs.GCSFsConfig{
+			Bucket:         u.FsConfig.GCSConfig.Bucket,
+			CredentialFile: u.FsConfig.GCSConfig.CredentialFile,
+			StorageClass:   u.FsConfig.GCSConfig.StorageClass,
+			KeyPrefix:      u.FsConfig.GCSConfig.KeyPrefix,
+		},
 	}
 
 	return User{
@@ -458,3 +472,7 @@ func (u *User) getNotificationFieldsAsEnvVars(action string) []string {
 		fmt.Sprintf("SFTPGO_USER_UID=%v", u.UID),
 		fmt.Sprintf("SFTPGO_USER_GID=%v", u.GID)}
 }
+
+func (u *User) getGCSCredentialsFilePath() string {
+	return filepath.Join(credentialsDirPath, fmt.Sprintf("%v_gcs_credentials.json", u.Username))
+}

docker/sftpgo/alpine/README.md

@@ -12,6 +12,7 @@ sudo groupadd -g 1003 sftpgrp && \
   curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20190828.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
   curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20191112.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
   curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20191230.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
+  curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20200116.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
   curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sftpgo.json -o /home/sftpuser/conf/sftpgo.json
 
 # Get and build SFTPGo image

go.mod

@@ -3,6 +3,8 @@ module github.com/drakkan/sftpgo
 go 1.13
 
 require (
+	cloud.google.com/go v0.52.0 // indirect
+	cloud.google.com/go/storage v1.5.0
 	github.com/alexedwards/argon2id v0.0.0-20190612080829-01a59b2b8802
 	github.com/aws/aws-sdk-go v1.28.3
 	github.com/cenkalti/backoff v2.2.1+incompatible // indirect
@@ -10,6 +12,7 @@ require (
 	github.com/go-chi/chi v4.0.2+incompatible
 	github.com/go-chi/render v1.0.1
 	github.com/go-sql-driver/mysql v1.5.0
+	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
 	github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c
 	github.com/lib/pq v1.3.0
 	github.com/mattn/go-sqlite3 v2.0.2+incompatible
@@ -23,7 +26,11 @@ require (
 	github.com/spf13/viper v1.6.1
 	go.etcd.io/bbolt v1.3.3
 	golang.org/x/crypto v0.0.0-20200109152110-61a87790db17
-	golang.org/x/sys v0.0.0-20191220142924-d4481acd189f
+	golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a // indirect
+	golang.org/x/sys v0.0.0-20200122134326-e047566fdf82
+	golang.org/x/tools v0.0.0-20200124200720-1b668f209185 // indirect
+	google.golang.org/api v0.15.0
+	google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150 // indirect
 	gopkg.in/natefinch/lumberjack.v2 v2.0.0
 )
 

go.sum

@@ -1,5 +1,26 @@
 cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
+cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
+cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
+cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
+cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
+cloud.google.com/go v0.50.0 h1:0E3eE8MX426vUOs7aHfI7aN1BrIzzzf4ccKCSfSjGmc=
+cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
+cloud.google.com/go v0.52.0 h1:GGslhk/BU052LPlnI1vpp3fcbUs+hQ3E+Doti/3/vF8=
+cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
+cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
+cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
+cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
+cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
+cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
+cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
+cloud.google.com/go/storage v1.5.0 h1:RPUcBvDeYgQFMfQu1eBMq6piD1SXmLH+vK3qjewZPus=
+cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
+dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
+github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
 github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
@@ -16,10 +37,14 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=
 github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
+github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
 github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
 github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
 github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
 github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
+github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
+github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
 github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
 github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
@@ -34,6 +59,8 @@ github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZm
 github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
 github.com/drakkan/pipeat v0.0.0-20200123131427-11c048cfc0ec h1:DXfzg1NXoesnFzdCyyi2uU3o1o0XiWTN2ZcpWDE7MCk=
 github.com/drakkan/pipeat v0.0.0-20200123131427-11c048cfc0ec/go.mod h1:wNYvIpR5rIhoezOYcpxcXz4HbIEOu7A45EqlQCA+h+w=
+github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
+github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@@ -41,6 +68,8 @@ github.com/go-chi/chi v4.0.2+incompatible h1:maB6vn6FqCxrpz4FqWdh4+lwpyZIQS7YEAU
 github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ=
 github.com/go-chi/render v1.0.1 h1:4/5tis2cKaNdnv9zFLfXzcquC9HbeZgCnxGnKrltBS8=
 github.com/go-chi/render v1.0.1/go.mod h1:pq4Rr7HbnsdaeHagklXub+p6Wd16Af5l9koip1OvJns=
+github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
+github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
 github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
@@ -52,15 +81,35 @@ github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7a
 github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
 github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
+github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
+github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
 github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
 github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
+github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
+github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
+github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
+github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
+github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
+github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
+github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
+github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
+github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
+github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
+github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
 github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
 github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
 github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c h1:svzQzfVE9t7Y1CGULS5PsMWs4/H4Au/ZTJzU/0CKgqc=
@@ -68,14 +117,20 @@ github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c/go.mod h1:YjKB0W
 github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
 github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
 github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
+github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
+github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=
 github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
+github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfEUpgAwUN0o=
+github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
 github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
 github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
@@ -124,6 +179,7 @@ github.com/prometheus/client_golang v1.3.0 h1:miYCvYqFXtl/J9FIy8eNpBfYthAEFg+Ys0
 github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeDPbaTKGT+JTgUa3og=
 github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
 github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/client_model v0.1.0 h1:ElTg5tNp4DqfV7UQjDqv2+RJlNzsDtvNAWccbItceIE=
 github.com/prometheus/client_model v0.1.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
@@ -138,6 +194,7 @@ github.com/prometheus/procfs v0.0.8 h1:+fpWZdT24pJBiqJdAwYBjPSk+5YmQzYNPYzQsdzLk
 github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
 github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
 github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
+github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
 github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
 github.com/rs/zerolog v1.17.2 h1:RMRHFw2+wF7LO0QqtELQwo8hqSmqISyCJeFeAAuWcRo=
@@ -178,34 +235,80 @@ github.com/zenazn/goji v0.9.0/go.mod h1:7S9M489iMyHBNxwZnk9/EHS098H4/F6TATF2mIxt
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
 go.etcd.io/bbolt v1.3.3 h1:MUGmc65QhB3pIlaQ5bB4LwqSj6GIonVJXpZiaKNyaKk=
 go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
+go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
+go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
+go.opencensus.io v0.22.2 h1:75k/FF0Q2YM8QYo07VPddOLBslDt1MZOdEslOHvmzAs=
+go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
 go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200109152110-61a87790db17 h1:nVJ3guKA9qdkEQ3TUdXI9QSINo2CUPM/cySEvw2w8I0=
 golang.org/x/crypto v0.0.0-20200109152110-61a87790db17/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
+golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
+golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
+golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
+golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
+golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
+golang.org/x/exp v0.0.0-20191227195350-da58074b4299 h1:zQpM52jfKHG6II1ISZY1ZcpygvuSFZpLwfluuF89XOg=
+golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
+golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a h1:7Wlg8L54In96HTWOaI4sreLJ6qfyGuvSau5el3fK41Y=
+golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
+golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
+golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
 golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
+golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
+golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
 golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f h1:J5lckAjkw6qYlOZNj90mLYNTEKDvWeuc1yieZ8qUzUE=
+golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
+golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
+golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
+golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
+golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
 golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
+golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
+golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190923162816-aa69164e4478 h1:l5EDrHhldLYb3ZRHDUhXF7Om7MvYXnkV9/iQNo1lX6g=
 golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 h1:efeOvDhwQ29Dj3SdAV/MJf8oukgn+8D8WgaCaRMchF8=
+golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa h1:F+8P+gmewFQYRk6JoLQLwjBCTu3mcIURZfNkVweuRKA=
+golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 h1:pE8b58s1HRDMi8RDc79m0HISf9D4TzseP40cEA6IGfs=
+golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
+golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -214,29 +317,103 @@ golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5h
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191220142924-d4481acd189f h1:68K/z8GLUxV76xGSqwTWw2gyk/jwn79LUL43rES2g8o=
 golang.org/x/sys v0.0.0-20191220142924-d4481acd189f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8 h1:JA8d3MPx/IToSyXZG/RhwYEtfrKO1Fxrqe8KrkiLXKM=
+golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200122134326-e047566fdf82 h1:ywK/j/KkyTHcdyYSZNXGjMwgmDSfjglYZ3vStQ/gSCU=
+golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
+golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
+golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
+golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20190828213141-aed303cbaa74/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4 h1:Toz2IK7k8rbltAXwNAxKcn9OzqyNfMUhUNjz3sL0NMk=
+golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/tools v0.0.0-20200124200720-1b668f209185 h1:UhyNb/h6VU6sPOVb6x118tYR91HRLBUtWS01bOSFAz4=
+golang.org/x/tools v0.0.0-20200124200720-1b668f209185/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
+google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
+google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
+google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
+google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
+google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
+google.golang.org/api v0.15.0 h1:yzlyyDW/J0w8yNFJIhiAJy4kq74S+1DOLdawELNxFMA=
+google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
+google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
+google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
+google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
 google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
+google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
+google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
+google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
+google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
+google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
+google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
+google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
+google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
+google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
+google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
+google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb h1:ADPHZzpzM4tk4V4S5cnCrr5SwzvlrPRmqqCuJDB8UTs=
+google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
+google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
+google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150 h1:VPpdpQkGvFicX9yo4G5oxZPi9ALBnEOZblPSa/Wa2m4=
+google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
+google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
 google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
+google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
+google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
+google.golang.org/grpc v1.26.0 h1:2dTRdpdFEEhJYQD8EMLB61nnrzSCTbG38PhqdhvOltg=
+google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
 gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
 gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
@@ -248,3 +425,9 @@ gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
+honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
+rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=

httpd/api_quota.go

@@ -14,6 +14,7 @@ func getQuotaScans(w http.ResponseWriter, r *http.Request) {
 }
 
 func startQuotaScan(w http.ResponseWriter, r *http.Request) {
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
 	var u dataprovider.User
 	err := render.DecodeJSON(r.Body, &u)
 	if err != nil {

httpd/api_user.go

@@ -73,6 +73,7 @@ func getUserByID(w http.ResponseWriter, r *http.Request) {
 }
 
 func addUser(w http.ResponseWriter, r *http.Request) {
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
 	var user dataprovider.User
 	err := render.DecodeJSON(r.Body, &user)
 	if err != nil {
@@ -93,6 +94,7 @@ func addUser(w http.ResponseWriter, r *http.Request) {
 }
 
 func updateUser(w http.ResponseWriter, r *http.Request) {
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
 	userID, err := strconv.ParseInt(chi.URLParam(r, "userID"), 10, 64)
 	if err != nil {
 		err = errors.New("Invalid userID")
@@ -100,10 +102,10 @@ func updateUser(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	user, err := dataprovider.GetUserByID(dataProvider, userID)
-	oldPermissions := user.Permissions
-	oldS3AccessSecret := ""
+	currentPermissions := user.Permissions
+	currentS3AccessSecret := ""
 	if user.FsConfig.Provider == 1 {
-		oldS3AccessSecret = user.FsConfig.S3Config.AccessSecret
+		currentS3AccessSecret = user.FsConfig.S3Config.AccessSecret
 	}
 	user.Permissions = make(map[string][]string)
 	if _, ok := err.(*dataprovider.RecordNotFoundError); ok {
@@ -120,13 +122,13 @@ func updateUser(w http.ResponseWriter, r *http.Request) {
 	}
 	// we use new Permissions if passed otherwise the old ones
 	if len(user.Permissions) == 0 {
-		user.Permissions = oldPermissions
+		user.Permissions = currentPermissions
 	}
 	// we use the new access secret if different from the old one and not empty
 	if user.FsConfig.Provider == 1 {
-		if utils.RemoveDecryptionKey(oldS3AccessSecret) == user.FsConfig.S3Config.AccessSecret ||
+		if utils.RemoveDecryptionKey(currentS3AccessSecret) == user.FsConfig.S3Config.AccessSecret ||
 			len(user.FsConfig.S3Config.AccessSecret) == 0 {
-			user.FsConfig.S3Config.AccessSecret = oldS3AccessSecret
+			user.FsConfig.S3Config.AccessSecret = currentS3AccessSecret
 		}
 	}
 	if user.ID != userID {

httpd/api_utils.go

@@ -439,6 +439,16 @@ func compareUserFsConfig(expected *dataprovider.User, actual *dataprovider.User)
 		expected.FsConfig.S3Config.KeyPrefix+"/" != actual.FsConfig.S3Config.KeyPrefix {
 		return errors.New("S3 key prefix mismatch")
 	}
+	if expected.FsConfig.GCSConfig.Bucket != actual.FsConfig.GCSConfig.Bucket {
+		return errors.New("GCS bucket mismatch")
+	}
+	if expected.FsConfig.GCSConfig.StorageClass != actual.FsConfig.GCSConfig.StorageClass {
+		return errors.New("GCS storage class mismatch")
+	}
+	if expected.FsConfig.GCSConfig.KeyPrefix != actual.FsConfig.GCSConfig.KeyPrefix &&
+		expected.FsConfig.GCSConfig.KeyPrefix+"/" != actual.FsConfig.GCSConfig.KeyPrefix {
+		return errors.New("GCS key prefix mismatch")
+	}
 	return nil
 }
 

httpd/httpd.go

@@ -33,6 +33,7 @@ const (
 	webConnectionsPath    = "/web/connections"
 	webStaticFilesPath    = "/static"
 	maxRestoreSize        = 10485760 // 10 MB
+	maxRequestSize        = 1048576  // 1MB
 )
 
 var (

httpd/httpd_test.go

@@ -3,9 +3,12 @@ package httpd_test
 import (
 	"bytes"
 	"crypto/rand"
+	"encoding/base64"
 	"encoding/json"
 	"fmt"
+	"io"
 	"io/ioutil"
+	"mime/multipart"
 	"net"
 	"net/http"
 	"net/http/httptest"
@@ -56,6 +59,7 @@ var (
 	defaultPerms       = []string{dataprovider.PermAny}
 	homeBasePath       string
 	backupsPath        string
+	credentialsPath    string
 	testServer         *httptest.Server
 	providerDriverName string
 )
@@ -66,7 +70,10 @@ func TestMain(m *testing.M) {
 	logger.InitLogger(logfilePath, 5, 1, 28, false, zerolog.DebugLevel)
 	config.LoadConfig(configDir, "")
 	providerConf := config.GetProviderConf()
+	credentialsPath = filepath.Join(os.TempDir(), "test_credentials")
+	providerConf.CredentialsPath = credentialsPath
 	providerDriverName = providerConf.Driver
+	os.RemoveAll(credentialsPath)
 
 	err := dataprovider.Initialize(providerConf, configDir)
 	if err != nil {
@@ -102,6 +109,7 @@ func TestMain(m *testing.M) {
 	exitCode := m.Run()
 	os.Remove(logfilePath)
 	os.RemoveAll(backupsPath)
+	os.RemoveAll(credentialsPath)
 	os.Exit(exitCode)
 }
 
@@ -250,6 +258,8 @@ func TestAddUserInvalidFsConfig(t *testing.T) {
 	if err != nil {
 		t.Errorf("unexpected error adding user with invalid fs config: %v", err)
 	}
+	os.RemoveAll(credentialsPath)
+	os.MkdirAll(credentialsPath, 0700)
 	u.FsConfig.S3Config.Bucket = "test"
 	u.FsConfig.S3Config.Region = "eu-west-1"
 	u.FsConfig.S3Config.AccessKey = "access-key"
@@ -261,6 +271,32 @@ func TestAddUserInvalidFsConfig(t *testing.T) {
 	if err != nil {
 		t.Errorf("unexpected error adding user with invalid fs config: %v", err)
 	}
+	u = getTestUser()
+	u.FsConfig.Provider = 2
+	u.FsConfig.GCSConfig.Bucket = ""
+	_, _, err = httpd.AddUser(u, http.StatusBadRequest)
+	if err != nil {
+		t.Errorf("unexpected error adding user with invalid fs config: %v", err)
+	}
+	u.FsConfig.GCSConfig.Bucket = "test"
+	u.FsConfig.GCSConfig.StorageClass = "Standard"
+	u.FsConfig.GCSConfig.KeyPrefix = "/somedir/subdir/"
+	u.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("test"))
+	_, _, err = httpd.AddUser(u, http.StatusBadRequest)
+	if err != nil {
+		t.Errorf("unexpected error adding user with invalid fs config: %v", err)
+	}
+	u.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir/"
+	u.FsConfig.GCSConfig.Credentials = ""
+	_, _, err = httpd.AddUser(u, http.StatusBadRequest)
+	if err != nil {
+		t.Errorf("unexpected error adding user with invalid fs config: %v", err)
+	}
+	u.FsConfig.GCSConfig.Credentials = "no base64 encoded"
+	_, _, err = httpd.AddUser(u, http.StatusBadRequest)
+	if err != nil {
+		t.Errorf("unexpected error adding user with invalid fs config: %v", err)
+	}
 }
 
 func TestUserPublicKey(t *testing.T) {
@@ -363,6 +399,56 @@ func TestUserS3Config(t *testing.T) {
 	}
 }
 
+func TestUserGCSConfig(t *testing.T) {
+	user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
+	if err != nil {
+		t.Errorf("unable to add user: %v", err)
+	}
+	os.RemoveAll(credentialsPath)
+	os.MkdirAll(credentialsPath, 0700)
+	user.FsConfig.Provider = 2
+	user.FsConfig.GCSConfig.Bucket = "test"
+	user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("fake credentials"))
+	user, _, err = httpd.UpdateUser(user, http.StatusOK)
+	if err != nil {
+		t.Errorf("unable to update user: %v", err)
+	}
+	_, err = httpd.RemoveUser(user, http.StatusOK)
+	if err != nil {
+		t.Errorf("unable to remove: %v", err)
+	}
+	user.Password = defaultPassword
+	user.ID = 0
+	// the user will be added since the credentials file is found
+	user, _, err = httpd.AddUser(user, http.StatusOK)
+	if err != nil {
+		t.Errorf("unable to add user: %v", err)
+	}
+	user.FsConfig.Provider = 1
+	user.FsConfig.S3Config.Bucket = "test1"
+	user.FsConfig.S3Config.Region = "us-east-1"
+	user.FsConfig.S3Config.AccessKey = "Server-Access-Key1"
+	user.FsConfig.S3Config.AccessSecret = "secret"
+	user.FsConfig.S3Config.Endpoint = "http://localhost:9000"
+	user.FsConfig.S3Config.KeyPrefix = "somedir/subdir"
+	user, _, err = httpd.UpdateUser(user, http.StatusOK)
+	if err != nil {
+		t.Errorf("unable to update user: %v", err)
+	}
+	user.FsConfig.Provider = 2
+	user.FsConfig.GCSConfig.Bucket = "test1"
+	user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("fake credentials"))
+	user, _, err = httpd.UpdateUser(user, http.StatusOK)
+	if err != nil {
+		t.Errorf("unable to update user: %v", err)
+	}
+
+	_, err = httpd.RemoveUser(user, http.StatusOK)
+	if err != nil {
+		t.Errorf("unable to remove: %v", err)
+	}
+}
+
 func TestUpdateUserNoCredentials(t *testing.T) {
 	user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
 	if err != nil {
@@ -594,6 +680,8 @@ func TestUserBaseDir(t *testing.T) {
 	dataprovider.Close(dataProvider)
 	config.LoadConfig(configDir, "")
 	providerConf = config.GetProviderConf()
+	providerConf.CredentialsPath = credentialsPath
+	os.RemoveAll(credentialsPath)
 	err = dataprovider.Initialize(providerConf, configDir)
 	if err != nil {
 		t.Errorf("error initializing data provider")
@@ -646,6 +734,8 @@ func TestProviderErrors(t *testing.T) {
 	os.Remove(backupFilePath)
 	config.LoadConfig(configDir, "")
 	providerConf := config.GetProviderConf()
+	providerConf.CredentialsPath = credentialsPath
+	os.RemoveAll(credentialsPath)
 	err = dataprovider.Initialize(providerConf, configDir)
 	if err != nil {
 		t.Errorf("error initializing data provider")
@@ -655,7 +745,17 @@ func TestProviderErrors(t *testing.T) {
 }
 
 func TestDumpdata(t *testing.T) {
-	_, _, err := httpd.Dumpdata("", http.StatusBadRequest)
+	dataProvider := dataprovider.GetProvider()
+	dataprovider.Close(dataProvider)
+	config.LoadConfig(configDir, "")
+	providerConf := config.GetProviderConf()
+	err := dataprovider.Initialize(providerConf, configDir)
+	if err != nil {
+		t.Errorf("error initializing data provider")
+	}
+	httpd.SetDataProvider(dataprovider.GetProvider())
+	sftpd.SetDataProvider(dataprovider.GetProvider())
+	_, _, err = httpd.Dumpdata("", http.StatusBadRequest)
 	if err != nil {
 		t.Errorf("unexpected error: %v", err)
 	}
@@ -680,6 +780,15 @@ func TestDumpdata(t *testing.T) {
 		}
 		os.Chmod(backupsPath, 0755)
 	}
+	providerConf = config.GetProviderConf()
+	providerConf.CredentialsPath = credentialsPath
+	os.RemoveAll(credentialsPath)
+	err = dataprovider.Initialize(providerConf, configDir)
+	if err != nil {
+		t.Errorf("error initializing data provider")
+	}
+	httpd.SetDataProvider(dataprovider.GetProvider())
+	sftpd.SetDataProvider(dataprovider.GetProvider())
 }
 
 func TestLoaddata(t *testing.T) {
@@ -1228,16 +1337,23 @@ func TestBasicWebUsersMock(t *testing.T) {
 	checkResponseCode(t, http.StatusBadRequest, rr.Code)
 	form := make(url.Values)
 	form.Set("username", user.Username)
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
+	b, contentType, _ := getMultipartFormData(form, "", "")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
-	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
+	b, contentType, _ = getMultipartFormData(form, "", "")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
-	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/0", strings.NewReader(form.Encode()))
+	b, contentType, _ = getMultipartFormData(form, "", "")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/0", &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusNotFound, rr.Code)
-	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/a", strings.NewReader(form.Encode()))
+	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/a", &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusBadRequest, rr.Code)
 	req, _ = http.NewRequest(http.MethodDelete, userPath+"/"+strconv.FormatInt(user.ID, 10), nil)
@@ -1261,90 +1377,103 @@ func TestWebUserAddMock(t *testing.T) {
 	form.Set("expiration_date", "")
 	form.Set("permissions", "*")
 	form.Set("sub_dirs_permissions", " /subdir:list ,download ")
+	b, contentType, _ := getMultipartFormData(form, "", "")
 	// test invalid url escape
-	req, _ := http.NewRequest(http.MethodPost, webUserPath+"?a=%2", strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ := http.NewRequest(http.MethodPost, webUserPath+"?a=%2", &b)
+	req.Header.Set("Content-Type", contentType)
 	rr := executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("public_keys", testPubKey)
 	form.Set("uid", strconv.FormatInt(int64(user.UID), 10))
 	form.Set("gid", "a")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid gid
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("gid", "0")
 	form.Set("max_sessions", "a")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid max sessions
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("max_sessions", "0")
 	form.Set("quota_size", "a")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid quota size
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("quota_size", "0")
 	form.Set("quota_files", "a")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid quota files
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("quota_files", "0")
 	form.Set("upload_bandwidth", "a")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid upload bandwidth
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("upload_bandwidth", strconv.FormatInt(user.UploadBandwidth, 10))
 	form.Set("download_bandwidth", "a")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid download bandwidth
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("download_bandwidth", strconv.FormatInt(user.DownloadBandwidth, 10))
 	form.Set("status", "a")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid status
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("status", strconv.Itoa(user.Status))
 	form.Set("expiration_date", "123")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid expiration date
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("expiration_date", "")
 	form.Set("allowed_ip", "invalid,ip")
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid allowed_ip
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("allowed_ip", "")
 	form.Set("denied_ip", "192.168.1.2") // it should be 192.168.1.2/32
+	b, contentType, _ = getMultipartFormData(form, "", "")
 	// test invalid denied_ip
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	form.Set("denied_ip", "")
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	b, contentType, _ = getMultipartFormData(form, "", "")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusSeeOther, rr.Code)
 	// the user already exists, was created with the above request
-	req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	b, contentType, _ = getMultipartFormData(form, "", "")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusOK, rr.Code)
 	req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@@ -1356,7 +1485,7 @@ func TestWebUserAddMock(t *testing.T) {
 		t.Errorf("Error decoding users: %v", err)
 	}
 	if len(users) != 1 {
-		t.Errorf("1 user is expected")
+		t.Errorf("1 user is expected, actual: %v", len(users))
 	}
 	newUser := users[0]
 	if newUser.UID != user.UID {
@@ -1413,8 +1542,9 @@ func TestWebUserUpdateMock(t *testing.T) {
 	form.Set("expiration_date", "2020-01-01 00:00:00")
 	form.Set("allowed_ip", " 192.168.1.3/32, 192.168.2.0/24 ")
 	form.Set("denied_ip", " 10.0.0.2/32 ")
-	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	b, contentType, _ := getMultipartFormData(form, "", "")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusSeeOther, rr.Code)
 	req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@@ -1504,8 +1634,9 @@ func TestWebUserS3Mock(t *testing.T) {
 	form.Set("s3_storage_class", user.FsConfig.S3Config.StorageClass)
 	form.Set("s3_endpoint", user.FsConfig.S3Config.Endpoint)
 	form.Set("s3_key_prefix", user.FsConfig.S3Config.KeyPrefix)
-	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	b, contentType, _ := getMultipartFormData(form, "", "")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
+	req.Header.Set("Content-Type", contentType)
 	rr = executeRequest(req)
 	checkResponseCode(t, http.StatusSeeOther, rr.Code)
 	req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@@ -1552,6 +1683,97 @@ func TestWebUserS3Mock(t *testing.T) {
 	checkResponseCode(t, http.StatusOK, rr.Code)
 }
 
+func TestWebUserGCSMock(t *testing.T) {
+	user := getTestUser()
+	userAsJSON := getUserAsJSON(t, user)
+	req, _ := http.NewRequest(http.MethodPost, userPath, bytes.NewBuffer(userAsJSON))
+	rr := executeRequest(req)
+	checkResponseCode(t, http.StatusOK, rr.Code)
+	err := render.DecodeJSON(rr.Body, &user)
+	if err != nil {
+		t.Errorf("Error get user: %v", err)
+	}
+	credentialsFilePath := filepath.Join(os.TempDir(), "gcs.json")
+	err = createTestFile(credentialsFilePath, 0)
+	if err != nil {
+		t.Errorf("unable to create credential test file: %v", err)
+	}
+	user.FsConfig.Provider = 2
+	user.FsConfig.GCSConfig.Bucket = "test"
+	user.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir/"
+	user.FsConfig.GCSConfig.StorageClass = "standard"
+	form := make(url.Values)
+	form.Set("username", user.Username)
+	form.Set("home_dir", user.HomeDir)
+	form.Set("uid", "0")
+	form.Set("gid", strconv.FormatInt(int64(user.GID), 10))
+	form.Set("max_sessions", strconv.FormatInt(int64(user.MaxSessions), 10))
+	form.Set("quota_size", strconv.FormatInt(user.QuotaSize, 10))
+	form.Set("quota_files", strconv.FormatInt(int64(user.QuotaFiles), 10))
+	form.Set("upload_bandwidth", "0")
+	form.Set("download_bandwidth", "0")
+	form.Set("permissions", "*")
+	form.Set("sub_dirs_permissions", "")
+	form.Set("status", strconv.Itoa(user.Status))
+	form.Set("expiration_date", "2020-01-01 00:00:00")
+	form.Set("allowed_ip", "")
+	form.Set("denied_ip", "")
+	form.Set("fs_provider", "2")
+	form.Set("gcs_bucket", user.FsConfig.GCSConfig.Bucket)
+	form.Set("gcs_storage_class", user.FsConfig.GCSConfig.StorageClass)
+	form.Set("gcs_key_prefix", user.FsConfig.GCSConfig.KeyPrefix)
+	b, contentType, _ := getMultipartFormData(form, "", "")
+	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
+	req.Header.Set("Content-Type", contentType)
+	rr = executeRequest(req)
+	checkResponseCode(t, http.StatusOK, rr.Code)
+	b, contentType, _ = getMultipartFormData(form, "gcs_credential_file", credentialsFilePath)
+	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
+	req.Header.Set("Content-Type", contentType)
+	rr = executeRequest(req)
+	checkResponseCode(t, http.StatusOK, rr.Code)
+	err = createTestFile(credentialsFilePath, 4096)
+	if err != nil {
+		t.Errorf("unable to create credential test file: %v", err)
+	}
+	b, contentType, _ = getMultipartFormData(form, "gcs_credential_file", credentialsFilePath)
+	req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
+	req.Header.Set("Content-Type", contentType)
+	rr = executeRequest(req)
+	checkResponseCode(t, http.StatusSeeOther, rr.Code)
+	req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
+	rr = executeRequest(req)
+	checkResponseCode(t, http.StatusOK, rr.Code)
+	var users []dataprovider.User
+	err = render.DecodeJSON(rr.Body, &users)
+	if err != nil {
+		t.Errorf("Error decoding users: %v", err)
+	}
+	if len(users) != 1 {
+		t.Errorf("1 user is expected")
+	}
+	updateUser := users[0]
+	if updateUser.ExpirationDate != 1577836800000 {
+		t.Errorf("invalid expiration date: %v", updateUser.ExpirationDate)
+	}
+	if updateUser.FsConfig.Provider != user.FsConfig.Provider {
+		t.Error("fs provider mismatch")
+	}
+	if updateUser.FsConfig.GCSConfig.Bucket != user.FsConfig.GCSConfig.Bucket {
+		t.Error("GCS bucket mismatch")
+	}
+	if updateUser.FsConfig.GCSConfig.StorageClass != user.FsConfig.GCSConfig.StorageClass {
+		t.Error("GCS storage class mismatch")
+	}
+	if updateUser.FsConfig.GCSConfig.KeyPrefix != user.FsConfig.GCSConfig.KeyPrefix {
+		t.Error("GCS key prefix mismatch")
+	}
+	req, _ = http.NewRequest(http.MethodDelete, userPath+"/"+strconv.FormatInt(user.ID, 10), nil)
+	rr = executeRequest(req)
+	checkResponseCode(t, http.StatusOK, rr.Code)
+	os.Remove(credentialsFilePath)
+}
+
 func TestProviderClosedMock(t *testing.T) {
 	if providerDriverName == dataprovider.BoltDataProviderName {
 		t.Skip("skipping test provider errors for bolt provider")
@@ -1571,6 +1793,8 @@ func TestProviderClosedMock(t *testing.T) {
 	checkResponseCode(t, http.StatusInternalServerError, rr.Code)
 	config.LoadConfig(configDir, "")
 	providerConf := config.GetProviderConf()
+	providerConf.CredentialsPath = credentialsPath
+	os.RemoveAll(credentialsPath)
 	err := dataprovider.Initialize(providerConf, configDir)
 	if err != nil {
 		t.Errorf("error initializing data provider")
@@ -1644,9 +1868,38 @@ func createTestFile(path string, size int64) error {
 		os.MkdirAll(baseDir, 0777)
 	}
 	content := make([]byte, size)
-	_, err := rand.Read(content)
-	if err != nil {
-		return err
+	if size > 0 {
+		_, err := rand.Read(content)
+		if err != nil {
+			return err
+		}
 	}
 	return ioutil.WriteFile(path, content, 0666)
 }
+
+func getMultipartFormData(values url.Values, fileFieldName, filePath string) (bytes.Buffer, string, error) {
+	var b bytes.Buffer
+	w := multipart.NewWriter(&b)
+	for k, v := range values {
+		for _, s := range v {
+			if err := w.WriteField(k, s); err != nil {
+				return b, "", err
+			}
+		}
+	}
+	if len(fileFieldName) > 0 && len(filePath) > 0 {
+		fw, err := w.CreateFormFile(fileFieldName, filepath.Base(filePath))
+		if err != nil {
+			return b, "", err
+		}
+		f, err := os.Open(filePath)
+		if err != nil {
+			return b, "", err
+		}
+		if _, err = io.Copy(fw, f); err != nil {
+			return b, "", err
+		}
+	}
+	err := w.Close()
+	return b, w.FormDataContentType(), err
+}

httpd/internal_test.go

@@ -6,7 +6,9 @@ import (
 	"html/template"
 	"net/http"
 	"net/http/httptest"
+	"net/url"
 	"os"
+	"strings"
 	"testing"
 
 	"github.com/drakkan/sftpgo/dataprovider"
@@ -280,6 +282,38 @@ func TestCompareUserFsConfig(t *testing.T) {
 	if err == nil {
 		t.Errorf("S3 key prefix does not match")
 	}
+	expected.FsConfig.S3Config.KeyPrefix = ""
+	expected.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir"
+	err = compareUserFsConfig(expected, actual)
+	if err == nil {
+		t.Errorf("GCS key prefix does not match")
+	}
+	expected.FsConfig.GCSConfig.KeyPrefix = ""
+	expected.FsConfig.GCSConfig.Bucket = "bucket"
+	err = compareUserFsConfig(expected, actual)
+	if err == nil {
+		t.Errorf("GCS bucket does not match")
+	}
+	expected.FsConfig.GCSConfig.Bucket = ""
+	expected.FsConfig.GCSConfig.StorageClass = "Standard"
+	err = compareUserFsConfig(expected, actual)
+	if err == nil {
+		t.Errorf("GCS storage class does not match")
+	}
+	expected.FsConfig.GCSConfig.StorageClass = ""
+}
+
+func TestGCSWebInvalidFormFile(t *testing.T) {
+	form := make(url.Values)
+	form.Set("username", "test_username")
+	form.Set("fs_provider", "2")
+	req, _ := http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
+	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req.ParseForm()
+	_, err := getFsConfigFromUserPostFields(req)
+	if err != http.ErrNotMultipart {
+		t.Errorf("unexpected error: %v", err)
+	}
 }
 
 func TestApiCallsWithBadURL(t *testing.T) {

httpd/schema/openapi.yaml

@@ -2,7 +2,7 @@ openapi: 3.0.1
 info:
   title: SFTPGo
   description: 'SFTPGo REST API'
-  version: 1.5.0
+  version: 1.6.0
 
 servers:
 - url: /api/v1
@@ -743,6 +743,26 @@ components:
         - access_secret
       nullable: true
       description: S3 Compatible Object Storage configuration details
+    GCSConfig:
+      type: object
+      properties:
+        bucket:
+          type: string
+          minLength: 1
+        credentials:
+          type: string
+          format: byte
+          description: Google Cloud Storage JSON credentials base64 encoded. This field must be populated only when adding/updating an user. It will be always omitted, since there are sensitive data, when you search/get users. The credentials will be stored in the configured "credentials_path"
+        storage_class:
+          type: string
+        key_prefix:
+          type: string
+          description: key_prefix is similar to a chroot directory for a local filesystem. If specified the SFTP user will only see contents that starts with this prefix and so you can restrict access to a specific virtual folder. The prefix, if not empty, must not start with "/" and must end with "/". If empty the whole bucket contents will be available
+          example: folder/subfolder/
+      required:
+        - bucket
+      nullable: true
+      description: Google Cloud Storage configuration details
     FilesystemConfig:
       type: object
       properties:
@@ -751,12 +771,16 @@ components:
           enum:
             - 0
             - 1
+            - 2
           description: >
             Providers:
               * `0` - local filesystem
               * `1` - S3 Compatible Object Storage
+              * `2` - Google Cloud Storage
         s3config:
           $ref: '#/components/schemas/S3Config'
+        gcsconfig:
+          $ref: '#/components/schemas/GCSConfig'
       description: Storage filesystem details
     User:
       type: object

httpd/web.go

@@ -1,8 +1,11 @@
 package httpd
 
 import (
+	"encoding/base64"
+	"errors"
 	"fmt"
 	"html/template"
+	"io/ioutil"
 	"net/http"
 	"path/filepath"
 	"strconv"
@@ -224,7 +227,7 @@ func getFiltersFromUserPostFields(r *http.Request) dataprovider.UserFilters {
 	return filters
 }
 
-func getFsConfigFromUserPostFields(r *http.Request) dataprovider.Filesystem {
+func getFsConfigFromUserPostFields(r *http.Request) (dataprovider.Filesystem, error) {
 	var fs dataprovider.Filesystem
 	provider, err := strconv.Atoi(r.Form.Get("fs_provider"))
 	if err != nil {
@@ -239,13 +242,33 @@ func getFsConfigFromUserPostFields(r *http.Request) dataprovider.Filesystem {
 		fs.S3Config.Endpoint = r.Form.Get("s3_endpoint")
 		fs.S3Config.StorageClass = r.Form.Get("s3_storage_class")
 		fs.S3Config.KeyPrefix = r.Form.Get("s3_key_prefix")
+	} else if fs.Provider == 2 {
+		fs.GCSConfig.Bucket = r.Form.Get("gcs_bucket")
+		fs.GCSConfig.StorageClass = r.Form.Get("gcs_storage_class")
+		fs.GCSConfig.KeyPrefix = r.Form.Get("gcs_key_prefix")
+		credentials, _, err := r.FormFile("gcs_credential_file")
+		if err == http.ErrMissingFile {
+			return fs, nil
+		}
+		if err != nil {
+			return fs, err
+		}
+		defer credentials.Close()
+		fileBytes, err := ioutil.ReadAll(credentials)
+		if err != nil || len(fileBytes) == 0 {
+			if len(fileBytes) == 0 {
+				err = errors.New("credentials file size must be greater than 0")
+			}
+			return fs, err
+		}
+		fs.GCSConfig.Credentials = base64.StdEncoding.EncodeToString(fileBytes)
 	}
-	return fs
+	return fs, nil
 }
 
 func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
 	var user dataprovider.User
-	err := r.ParseForm()
+	err := r.ParseMultipartForm(maxRequestSize)
 	if err != nil {
 		return user, err
 	}
@@ -292,6 +315,10 @@ func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
 		}
 		expirationDateMillis = utils.GetTimeAsMsSinceEpoch(expirationDate)
 	}
+	fsConfig, err := getFsConfigFromUserPostFields(r)
+	if err != nil {
+		return user, err
+	}
 	user = dataprovider.User{
 		Username:          r.Form.Get("username"),
 		Password:          r.Form.Get("password"),
@@ -308,7 +335,7 @@ func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
 		Status:            status,
 		ExpirationDate:    expirationDateMillis,
 		Filters:           getFiltersFromUserPostFields(r),
-		FsConfig:          getFsConfigFromUserPostFields(r),
+		FsConfig:          fsConfig,
 	}
 	return user, err
 }
@@ -365,6 +392,7 @@ func handleWebUpdateUserGet(userID string, w http.ResponseWriter, r *http.Reques
 }
 
 func handleWebAddUserPost(w http.ResponseWriter, r *http.Request) {
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
 	user, err := getUserFromPostFields(r)
 	if err != nil {
 		renderAddUserPage(w, user, err.Error())
@@ -379,6 +407,7 @@ func handleWebAddUserPost(w http.ResponseWriter, r *http.Request) {
 }
 
 func handleWebUpdateUserPost(userID string, w http.ResponseWriter, r *http.Request) {
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
 	id, err := strconv.ParseInt(userID, 10, 64)
 	if err != nil {
 		renderBadRequestPage(w, err)

metrics/metrics.go

@@ -246,22 +246,94 @@ var (
 		Help: "The total number of successful S3 head bucket requests",
 	})
 
-	// totalS3CreateBucket is the metric that reports the total successful S3 create bucket requests
-	totalS3CreateBucket = promauto.NewCounter(prometheus.CounterOpts{
-		Name: "sftpgo_s3_create_bucket",
-		Help: "The total number of successful S3 create bucket requests",
-	})
-
 	// totalS3HeadBucketErrors is the metric that reports the total S3 head bucket errors
 	totalS3HeadBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
 		Name: "sftpgo_s3_head_bucket_errors",
 		Help: "The total number of S3 head bucket errors",
 	})
 
-	// totalS3CreateBucketErrors is the metric that reports the total S3 create bucket errors
-	totalS3CreateBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
-		Name: "sftpgo_s3_create_bucket_errors",
-		Help: "The total number of S3 create bucket errors",
+	// totalGCSUploads is the metric that reports the total number of successful GCS uploads
+	totalGCSUploads = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_uploads_total",
+		Help: "The total number of successful GCS uploads",
+	})
+
+	// totalGCSDownloads is the metric that reports the total number of successful GCS downloads
+	totalGCSDownloads = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_downloads_total",
+		Help: "The total number of successful GCS downloads",
+	})
+
+	// totalGCSUploadErrors is the metric that reports the total number of GCS upload errors
+	totalGCSUploadErrors = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_upload_errors_total",
+		Help: "The total number of GCS upload errors",
+	})
+
+	// totalGCSDownloadErrors is the metric that reports the total number of GCS download errors
+	totalGCSDownloadErrors = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_download_errors_total",
+		Help: "The total number of GCS download errors",
+	})
+
+	// totalGCSUploadSize is the metric that reports the total GCS uploads size as bytes
+	totalGCSUploadSize = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_upload_size",
+		Help: "The total GCS upload size as bytes, partial uploads are included",
+	})
+
+	// totalGCSDownloadSize is the metric that reports the total GCS downloads size as bytes
+	totalGCSDownloadSize = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_download_size",
+		Help: "The total GCS download size as bytes, partial downloads are included",
+	})
+
+	// totalS3ListObjects is the metric that reports the total successful GCS list objects requests
+	totalGCSListObjects = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_list_objects",
+		Help: "The total number of successful GCS list objects requests",
+	})
+
+	// totalGCSCopyObject is the metric that reports the total successful GCS copy object requests
+	totalGCSCopyObject = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_copy_object",
+		Help: "The total number of successful GCS copy object requests",
+	})
+
+	// totalGCSDeleteObject is the metric that reports the total successful S3 delete object requests
+	totalGCSDeleteObject = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_delete_object",
+		Help: "The total number of successful GCS delete object requests",
+	})
+
+	// totalGCSListObjectsError is the metric that reports the total GCS list objects errors
+	totalGCSListObjectsErrors = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_list_objects_errors",
+		Help: "The total number of GCS list objects errors",
+	})
+
+	// totalGCSCopyObjectErrors is the metric that reports the total GCS copy object errors
+	totalGCSCopyObjectErrors = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_copy_object_errors",
+		Help: "The total number of GCS copy object errors",
+	})
+
+	// totalGCSDeleteObjectErrors is the metric that reports the total GCS delete object errors
+	totalGCSDeleteObjectErrors = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_delete_object_errors",
+		Help: "The total number of GCS delete object errors",
+	})
+
+	// totalGCSHeadBucket is the metric that reports the total successful GCS head bucket requests
+	totalGCSHeadBucket = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_head_bucket",
+		Help: "The total number of successful GCS head bucket requests",
+	})
+
+	// totalGCSHeadBucketErrors is the metric that reports the total GCS head bucket errors
+	totalGCSHeadBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
+		Name: "sftpgo_gcs_head_bucket_errors",
+		Help: "The total number of GCS head bucket errors",
 	})
 )
 
@@ -343,12 +415,60 @@ func S3HeadBucketCompleted(err error) {
 	}
 }
 
-// S3CreateBucketCompleted updates metrics after an S3 create bucket request terminates
-func S3CreateBucketCompleted(err error) {
+// GCSTransferCompleted updates metrics after a GCS upload or a download
+func GCSTransferCompleted(bytes int64, transferKind int, err error) {
+	if transferKind == 0 {
+		// upload
+		if err == nil {
+			totalGCSUploads.Inc()
+		} else {
+			totalGCSUploadErrors.Inc()
+		}
+		totalGCSUploadSize.Add(float64(bytes))
+	} else {
+		// download
+		if err == nil {
+			totalGCSDownloads.Inc()
+		} else {
+			totalGCSDownloadErrors.Inc()
+		}
+		totalGCSDownloadSize.Add(float64(bytes))
+	}
+}
+
+// GCSListObjectsCompleted updates metrics after a GCS list objects request terminates
+func GCSListObjectsCompleted(err error) {
+	if err == nil {
+		totalGCSListObjects.Inc()
+	} else {
+		totalGCSListObjectsErrors.Inc()
+	}
+}
+
+// GCSCopyObjectCompleted updates metrics after a GCS copy object request terminates
+func GCSCopyObjectCompleted(err error) {
+	if err == nil {
+		totalGCSCopyObject.Inc()
+	} else {
+		totalGCSCopyObjectErrors.Inc()
+	}
+}
+
+// GCSDeleteObjectCompleted updates metrics after a GCS delete object request terminates
+func GCSDeleteObjectCompleted(err error) {
+	if err == nil {
+		totalGCSDeleteObject.Inc()
+	} else {
+		totalGCSDeleteObjectErrors.Inc()
+	}
+}
+
+// GCSHeadBucketCompleted updates metrics after a GCS head bucket request terminates
+func GCSHeadBucketCompleted(err error) {
 	if err == nil {
-		totalS3CreateBucket.Inc()
+		totalGCSHeadBucket.Inc()
 	} else {
-		totalS3CreateBucketErrors.Inc()
+		totalGCSHeadBucketErrors.Inc()
 	}
 }
 

scripts/README.md

@@ -54,6 +54,7 @@ Output:
   "download_bandwidth": 60,
   "expiration_date": 1546297200000,
   "filesystem": {
+    "gcsconfig": {},
     "provider": 1,
     "s3config": {
       "access_key": "accesskey",
@@ -139,6 +140,7 @@ Output:
   "download_bandwidth": 80,
   "expiration_date": 0,
   "filesystem": {
+    "gcsconfig": {},
     "provider": 0,
     "s3config": {}
   },
@@ -191,6 +193,7 @@ Output:
     "download_bandwidth": 80,
     "expiration_date": 0,
     "filesystem": {
+      "gcsconfig": {},
       "provider": 0,
       "s3config": {}
     },

scripts/sftpgo_api_cli.py

@@ -1,5 +1,6 @@
 #!/usr/bin/env python
 import argparse
+import base64
 from datetime import datetime
 import json
 import platform
@@ -74,7 +75,7 @@ class SFTPGoApiRequests:
 					max_sessions=0, quota_size=0, quota_files=0, permissions={}, upload_bandwidth=0, download_bandwidth=0,
 					status=1, expiration_date=0, allowed_ip=[], denied_ip=[], fs_provider='local', s3_bucket='',
 					s3_region='', s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='',
-					s3_key_prefix=''):
+					s3_key_prefix='', gcs_bucket='', gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
 		user = {"id":user_id, "username":username, "uid":uid, "gid":gid,
 			"max_sessions":max_sessions, "quota_size":quota_size, "quota_files":quota_files,
 			"upload_bandwidth":upload_bandwidth, "download_bandwidth":download_bandwidth,
@@ -92,9 +93,9 @@ class SFTPGoApiRequests:
 			user.update({"permissions":permissions})
 		if allowed_ip or denied_ip:
 			user.update({"filters":self.buildFilters(allowed_ip, denied_ip)})
-		user.update({"filesystem":self.buildFsConfig(fs_provider, s3_bucket, s3_region, s3_access_key,
-														s3_access_secret, s3_endpoint, s3_storage_class,
-														s3_key_prefix)})
+		user.update({"filesystem":self.buildFsConfig(fs_provider, s3_bucket, s3_region, s3_access_key, s3_access_secret,
+													s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket,
+													gcs_key_prefix, gcs_storage_class, gcs_credentials_file)})
 		return user
 
 	def buildPermissions(self, root_perms, subdirs_perms):
@@ -129,13 +130,19 @@ class SFTPGoApiRequests:
 		return filters
 
 	def buildFsConfig(self, fs_provider, s3_bucket, s3_region, s3_access_key, s3_access_secret, s3_endpoint,
-					s3_storage_class, s3_key_prefix):
+					s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class, gcs_credentials_file):
 		fs_config = {'provider':0}
 		if fs_provider == 'S3':
 			s3config = {'bucket':s3_bucket, 'region':s3_region, 'access_key':s3_access_key, 'access_secret':
 					s3_access_secret, 'endpoint':s3_endpoint, 'storage_class':s3_storage_class, 'key_prefix':
 					s3_key_prefix}
 			fs_config.update({'provider':1, 's3config':s3config})
+		elif fs_provider == 'GCS':
+			gcsconfig = {'bucket':gcs_bucket, 'key_prefix':gcs_key_prefix, 'storage_class':gcs_storage_class}
+			if gcs_credentials_file:
+				with open(gcs_credentials_file) as creds:
+					gcsconfig.update({'credentials':base64.b64encode(creds.read().encode('UTF-8')).decode('UTF-8')})
+			fs_config.update({'provider':2, 'gcsconfig':gcsconfig})
 		return fs_config
 
 	def getUsers(self, limit=100, offset=0, order="ASC", username=""):
@@ -150,11 +157,13 @@ class SFTPGoApiRequests:
 	def addUser(self, username="", password="", public_keys="", home_dir="", uid=0, gid=0, max_sessions=0, quota_size=0,
 			quota_files=0, perms=[], upload_bandwidth=0, download_bandwidth=0, status=1, expiration_date=0,
 			subdirs_permissions=[], allowed_ip=[], denied_ip=[], fs_provider='local', s3_bucket='', s3_region='',
-			s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='', s3_key_prefix=''):
+			s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='', s3_key_prefix='', gcs_bucket='',
+			gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
 		u = self.buildUserObject(0, username, password, public_keys, home_dir, uid, gid, max_sessions,
 			quota_size, quota_files, self.buildPermissions(perms, subdirs_permissions), upload_bandwidth, download_bandwidth,
-			status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region,
-			s3_access_key, s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix)
+			status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region, s3_access_key,
+			s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class,
+			gcs_credentials_file)
 		r = requests.post(self.userPath, json=u, auth=self.auth, verify=self.verify)
 		self.printResponse(r)
 
@@ -162,11 +171,12 @@ class SFTPGoApiRequests:
 				quota_size=0, quota_files=0, perms=[], upload_bandwidth=0, download_bandwidth=0, status=1,
 				expiration_date=0, subdirs_permissions=[], allowed_ip=[], denied_ip=[], fs_provider='local',
 				s3_bucket='', s3_region='', s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='',
-				s3_key_prefix=''):
+				s3_key_prefix='', gcs_bucket='', gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
 		u = self.buildUserObject(user_id, username, password, public_keys, home_dir, uid, gid, max_sessions,
 			quota_size, quota_files, self.buildPermissions(perms, subdirs_permissions), upload_bandwidth, download_bandwidth,
 			status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region, s3_access_key,
-			s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix)
+			s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class,
+			gcs_credentials_file)
 		r = requests.put(urlparse.urljoin(self.userPath, "user/" + str(user_id)), json=u, auth=self.auth, verify=self.verify)
 		self.printResponse(r)
 
@@ -420,7 +430,7 @@ def addCommonUserArguments(parser):
 					help='Allowed IP/Mask in CIDR notation. For example "192.168.2.0/24" or "2001:db8::/32". Default: %(default)s')
 	parser.add_argument('-N', '--denied-ip', type=str, nargs='+', default=[],
 					help='Denied IP/Mask in CIDR notation. For example "192.168.2.0/24" or "2001:db8::/32". Default: %(default)s')
-	parser.add_argument('--fs', type=str, default='local', choices=['local', 'S3'],
+	parser.add_argument('--fs', type=str, default='local', choices=['local', 'S3', 'GCS'],
 					help='Filesystem provider. Default: %(default)s')
 	parser.add_argument('--s3-bucket', type=str, default='', help='Default: %(default)s')
 	parser.add_argument('--s3-key-prefix', type=str, default='', help='Virtual root directory. If non empty only this ' +
@@ -431,6 +441,12 @@ def addCommonUserArguments(parser):
 	parser.add_argument('--s3-access-secret', type=str, default='', help='Default: %(default)s')
 	parser.add_argument('--s3-endpoint', type=str, default='', help='Default: %(default)s')
 	parser.add_argument('--s3-storage-class', type=str, default='', help='Default: %(default)s')
+	parser.add_argument('--gcs-bucket', type=str, default='', help='Default: %(default)s')
+	parser.add_argument('--gcs-key-prefix', type=str, default='', help='Virtual root directory. If non empty only this ' +
+					'directory and its contents will be available. Cannot start with "/". For example "folder/subfolder/".' +
+					' Default: %(default)s')
+	parser.add_argument('--gcs-storage-class', type=str, default='', help='Default: %(default)s')
+	parser.add_argument('--gcs-credentials-file', type=str, default='', help='Default: %(default)s')
 
 
 if __name__ == '__main__':
@@ -534,14 +550,16 @@ if __name__ == '__main__':
 				args.quota_size, args.quota_files, args.permissions, args.upload_bandwidth, args.download_bandwidth,
 				args.status, getDatetimeAsMillisSinceEpoch(args.expiration_date), args.subdirs_permissions, args.allowed_ip,
 				args.denied_ip, args.fs, args.s3_bucket, args.s3_region, args.s3_access_key, args.s3_access_secret,
-				args.s3_endpoint, args.s3_storage_class, args.s3_key_prefix)
+				args.s3_endpoint, args.s3_storage_class, args.s3_key_prefix, args.gcs_bucket, args.gcs_key_prefix,
+				args.gcs_storage_class, args.gcs_credentials_file)
 	elif args.command == 'update-user':
 		api.updateUser(args.id, args.username, args.password, args.public_keys, args.home_dir, args.uid, args.gid,
 					args.max_sessions, args.quota_size, args.quota_files, args.permissions, args.upload_bandwidth,
 					args.download_bandwidth, args.status, getDatetimeAsMillisSinceEpoch(args.expiration_date),
 					args.subdirs_permissions, args.allowed_ip, args.denied_ip, args.fs, args.s3_bucket, args.s3_region,
 					args.s3_access_key, args.s3_access_secret, args.s3_endpoint, args.s3_storage_class,
-					args.s3_key_prefix)
+					args.s3_key_prefix, args.gcs_bucket, args.gcs_key_prefix, args.gcs_storage_class,
+					args.gcs_credentials_file)
 	elif args.command == 'delete-user':
 		api.deleteUser(args.id)
 	elif args.command == 'get-users':

service/service.go

@@ -6,6 +6,7 @@ import (
 	"math/rand"
 	"os"
 	"os/signal"
+	"path/filepath"
 	"strings"
 	"syscall"
 	"time"
@@ -146,6 +147,7 @@ func (s *Service) StartPortableMode(sftpdPort int, enabledSSHCommands []string,
 	}
 	dataProviderConf := config.GetProviderConf()
 	dataProviderConf.Driver = dataprovider.MemoryDataProviderName
+	dataProviderConf.CredentialsPath = filepath.Join(os.TempDir(), "credentials")
 	config.SetProviderConf(dataProviderConf)
 	httpdConf := config.GetHTTPDConfig()
 	httpdConf.BindPort = 0

sftpd/sftpd_test.go

@@ -1053,7 +1053,7 @@ func TestLoginInvalidFs(t *testing.T) {
 		t.Errorf("unable to add user: %v", err)
 	}
 	// we update the database using sqlite3 CLI since we cannot add an user with an invalid config
-	time.Sleep(150 * time.Millisecond)
+	time.Sleep(200 * time.Millisecond)
 	updateUserQuery := fmt.Sprintf("UPDATE users SET filesystem='{\"provider\":1}' WHERE id=%v", user.ID)
 	cmd := exec.Command("sqlite3", dbPath, updateUserQuery)
 	out, err := cmd.CombinedOutput()
@@ -3039,7 +3039,11 @@ func TestRelativePaths(t *testing.T) {
 		KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
 	}
 	s3fs, _ := vfs.NewS3Fs("", user.GetHomeDir(), s3config)
-	filesystems = append(filesystems, s3fs)
+	gcsConfig := vfs.GCSFsConfig{
+		KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
+	}
+	gcsfs, _ := vfs.NewGCSFs("", user.GetHomeDir(), gcsConfig)
+	filesystems = append(filesystems, s3fs, gcsfs)
 	for _, fs := range filesystems {
 		path = filepath.Join(user.HomeDir, "/")
 		rel = fs.GetRelativePath(path)
@@ -3104,7 +3108,11 @@ func TestResolvePaths(t *testing.T) {
 	}
 	os.MkdirAll(user.GetHomeDir(), 0777)
 	s3fs, _ := vfs.NewS3Fs("", user.GetHomeDir(), s3config)
-	filesystems = append(filesystems, s3fs)
+	gcsConfig := vfs.GCSFsConfig{
+		KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
+	}
+	gcsfs, _ := vfs.NewGCSFs("", user.GetHomeDir(), gcsConfig)
+	filesystems = append(filesystems, s3fs, gcsfs)
 	for _, fs := range filesystems {
 		path = "/"
 		resolved, _ = fs.ResolvePath(filepath.ToSlash(path))
@@ -3509,7 +3517,8 @@ func TestSCPBasicHandling(t *testing.T) {
 		t.Errorf("stat for the downloaded file must succeed")
 	} else {
 		if fi.Size() != testFileSize {
-			t.Errorf("size of the file downloaded via SCP does not match the expected one")
+			t.Errorf("size of the file downloaded via SCP does not match the expected one: %v/%v",
+				fi.Size(), testFileSize)
 		}
 	}
 	os.Remove(localPath)
@@ -3582,7 +3591,8 @@ func TestSCPUploadFileOverwrite(t *testing.T) {
 		t.Errorf("stat for the downloaded file must succeed")
 	} else {
 		if fi.Size() != testFileSize {
-			t.Errorf("size of the file downloaded via SCP does not match the expected one")
+			t.Errorf("size of the file downloaded via SCP does not match the expected one: %v/%v",
+				fi.Size(), testFileSize)
 		}
 	}
 	os.Remove(localPath)

sftpgo.json

@@ -42,7 +42,8 @@
       "http_notification_url": ""
     },
     "external_auth_program": "",
-    "external_auth_scope": 0
+    "external_auth_scope": 0,
+    "credentials_path": "credentials"
   },
   "httpd": {
     "bind_port": 8080,

templates/user.html

@@ -15,7 +15,7 @@
     <div class="card-body text-form-error">{{.Error}}</div>
 </div>
 {{end}}
-<form id="user_form" action="{{.CurrentURL}}" method="POST" autocomplete="off">
+<form id="user_form" enctype="multipart/form-data" action="{{.CurrentURL}}" method="POST" autocomplete="off">
     <div class="form-group row">
         <label for="idUsername" class="col-sm-2 col-form-label">Username</label>
         <div class="col-sm-10">
@@ -194,14 +194,15 @@
     <div class="form-group row">
         <label for="idFilesystem" class="col-sm-2 col-form-label">Storage</label>
         <div class="col-sm-10">
-            <select class="form-control" id="idFilesystem" name="fs_provider">
+            <select class="form-control" id="idFilesystem" name="fs_provider" onchange="onFilesystemChanged(this.value)">
                 <option value="0" {{if eq .User.FsConfig.Provider 0 }}selected{{end}}>local</option>
-                <option value="1" {{if eq .User.FsConfig.Provider 1 }}selected{{end}}>S3</option>
+                <option value="1" {{if eq .User.FsConfig.Provider 1 }}selected{{end}}>Amazon S3 (Compatible)</option>
+                <option value="2" {{if eq .User.FsConfig.Provider 2 }}selected{{end}}>Google Cloud Storage</option>
             </select>
         </div>
     </div>
 
-    <div class="form-group row">
+    <div class="form-group row s3">
         <label for="idS3Bucket" class="col-sm-2 col-form-label">S3 Bucket</label>
         <div class="col-sm-3">
             <input type="text" class="form-control" id="idS3Bucket" name="s3_bucket" placeholder=""
@@ -215,7 +216,7 @@
         </div>
     </div>
 
-    <div class="form-group row">
+    <div class="form-group row s3">
         <label for="idS3AccessKey" class="col-sm-2 col-form-label">S3 Access Key</label>
         <div class="col-sm-3">
             <input type="text" class="form-control" id="idS3AccessKey" name="s3_access_key" placeholder=""
@@ -229,7 +230,7 @@
         </div>
     </div>
 
-    <div class="form-group row">
+    <div class="form-group row s3">
         <label for="idS3StorageClass" class="col-sm-2 col-form-label">S3 Storage Class</label>
         <div class="col-sm-3">
             <input type="text" class="form-control" id="idS3StorageClass" name="s3_storage_class" placeholder=""
@@ -243,7 +244,7 @@
         </div>
     </div>
 
-    <div class="form-group row">
+    <div class="form-group row s3">
         <label for="idS3KeyPrefix" class="col-sm-2 col-form-label">S3 Key Prefix</label>
         <div class="col-sm-10">
             <input type="text" class="form-control" id="idS3KeyPrefix" name="s3_key_prefix" placeholder=""
@@ -254,6 +255,43 @@
         </div>
     </div>
 
+    <div class="form-group row gcs">
+        <label for="idGCSBucket" class="col-sm-2 col-form-label">GCS Bucket</label>
+        <div class="col-sm-10">
+            <input type="text" class="form-control" id="idGCSBucket" name="gcs_bucket" placeholder=""
+                value="{{.User.FsConfig.GCSConfig.Bucket}}" maxlength="255">
+        </div>
+    </div>
+
+    <div class="form-group row gcs">
+        <label for="idGCSCredentialFile" class="col-sm-2 col-form-label">GCS Credential file</label>
+        <div class="col-sm-4">
+            <input type="file" class="form-control-file" id="idGCSCredentialFile" name="gcs_credential_file"
+                aria-describedby="GCSCredentialsHelpBlock">
+            <small id="GCSCredentialsHelpBlock" class="form-text text-muted">
+                Add or update credentials from a JSON file
+            </small>
+        </div>
+        <div class="col-sm-1"></div>
+        <label for="idGCSStorageClass" class="col-sm-2 col-form-label">GCS Storage Class</label>
+        <div class="col-sm-3">
+            <input type="text" class="form-control" id="idGCSStorageClass" name="gcs_storage_class" placeholder=""
+                value="{{.User.FsConfig.GCSConfig.StorageClass}}" maxlength="255">
+        </div>
+    </div>
+
+    <div class="form-group row gcs">
+        <label for="idGCSKeyPrefix" class="col-sm-2 col-form-label">GCS Key Prefix</label>
+        <div class="col-sm-10">
+            <input type="text" class="form-control" id="idGCSKeyPrefix" name="gcs_key_prefix" placeholder=""
+                value="{{.User.FsConfig.GCSConfig.KeyPrefix}}" maxlength="255" aria-describedby="GCSKeyPrefixHelpBlock">
+            <small id="GCSKeyPrefixHelpBlock" class="form-text text-muted">
+                Similar to a chroot for local filesystem. Cannot start with "/". Example: "somedir/subdir/".
+            </small>
+        </div>
+    </div>
+
+
     <input type="hidden" name="expiration_date" id="hidden_start_datetime" value="">
     <button type="submit" class="btn btn-primary float-right mt-3 mb-5 px-5 px-3">Submit</button>
 </form>
@@ -295,6 +333,22 @@
             }
             return true;
         });
+
+        onFilesystemChanged('{{.User.FsConfig.Provider}}');
+
     });
+
+    function onFilesystemChanged(val){
+        if (val == '1'){
+            $('.form-group.row.gcs').hide();
+            $('.form-group.row.s3').show();
+        } else if (val == '2'){
+            $('.form-group.row.gcs').show();
+            $('.form-group.row.s3').hide();
+        } else {
+            $('.form-group.row.gcs').hide();
+            $('.form-group.row.s3').hide();
+        }
+    }
 </script>
 {{end}}

vfs/s3fileinfo.go → vfs/fileinfo.go

@@ -5,8 +5,8 @@ import (
 	"time"
 )
 
-// S3FileInfo implements os.FileInfo for a file in S3.
-type S3FileInfo struct {
+// FileInfo implements os.FileInfo for a file in S3.
+type FileInfo struct {
 	name        string
 	sizeInBytes int64
 	modTime     time.Time
@@ -14,14 +14,14 @@ type S3FileInfo struct {
 	sys         interface{}
 }
 
-// NewS3FileInfo creates file info.
-func NewS3FileInfo(name string, isDirectory bool, sizeInBytes int64, modTime time.Time) S3FileInfo {
+// NewFileInfo creates file info.
+func NewFileInfo(name string, isDirectory bool, sizeInBytes int64, modTime time.Time) FileInfo {
 	mode := os.FileMode(0644)
 	if isDirectory {
 		mode = os.FileMode(0755) | os.ModeDir
 	}
 
-	return S3FileInfo{
+	return FileInfo{
 		name:        name,
 		sizeInBytes: sizeInBytes,
 		modTime:     modTime,
@@ -30,31 +30,31 @@ func NewS3FileInfo(name string, isDirectory bool, sizeInBytes int64, modTime tim
 }
 
 // Name provides the base name of the file.
-func (fi S3FileInfo) Name() string {
+func (fi FileInfo) Name() string {
 	return fi.name
 }
 
 // Size provides the length in bytes for a file.
-func (fi S3FileInfo) Size() int64 {
+func (fi FileInfo) Size() int64 {
 	return fi.sizeInBytes
 }
 
 // Mode provides the file mode bits
-func (fi S3FileInfo) Mode() os.FileMode {
+func (fi FileInfo) Mode() os.FileMode {
 	return fi.mode
 }
 
 // ModTime provides the last modification time.
-func (fi S3FileInfo) ModTime() time.Time {
+func (fi FileInfo) ModTime() time.Time {
 	return fi.modTime
 }
 
 // IsDir provides the abbreviation for Mode().IsDir()
-func (fi S3FileInfo) IsDir() bool {
+func (fi FileInfo) IsDir() bool {
 	return fi.mode&os.ModeDir != 0
 }
 
 // Sys provides the underlying data source (can return nil)
-func (fi S3FileInfo) Sys() interface{} {
+func (fi FileInfo) Sys() interface{} {
 	return fi.getFileInfoSys()
 }

vfs/s3fileinfo_unix.go → vfs/fileinfo_unix.go

@@ -21,7 +21,7 @@ func init() {
 	}
 }
 
-func (fi S3FileInfo) getFileInfoSys() interface{} {
+func (fi FileInfo) getFileInfoSys() interface{} {
 	return &syscall.Stat_t{
 		Uid: uint32(defaultUID),
 		Gid: uint32(defaultGID)}

vfs/s3fileinfo_windows.go → vfs/fileinfo_windows.go

@@ -2,6 +2,6 @@ package vfs
 
 import "syscall"
 
-func (fi S3FileInfo) getFileInfoSys() interface{} {
+func (fi FileInfo) getFileInfoSys() interface{} {
 	return syscall.Win32FileAttributeData{}
 }

vfs/gcsfs.go

@@ -0,0 +1,511 @@
+package vfs
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path"
+	"strings"
+	"time"
+
+	"cloud.google.com/go/storage"
+	"github.com/drakkan/sftpgo/logger"
+	"github.com/drakkan/sftpgo/metrics"
+	"github.com/eikenb/pipeat"
+	"google.golang.org/api/googleapi"
+	"google.golang.org/api/iterator"
+	"google.golang.org/api/option"
+)
+
+var (
+	// we cannot use attrs selection until this bug is fixed:
+	//
+	// https://github.com/googleapis/google-cloud-go/issues/1763
+	//
+	gcsDefaultFieldsSelection = []string{"Name", "Size", "Deleted", "Updated"}
+)
+
+// GCSFsConfig defines the configuration for Google Cloud Storage based filesystem
+type GCSFsConfig struct {
+	Bucket string `json:"bucket,omitempty"`
+	// KeyPrefix is similar to a chroot directory for local filesystem.
+	// If specified the SFTP user will only see objects that starts with
+	// this prefix and so you can restrict access to a specific virtual
+	// folder. The prefix, if not empty, must not start with "/" and must
+	// end with "/".
+	// If empty the whole bucket contents will be available
+	KeyPrefix      string `json:"key_prefix,omitempty"`
+	CredentialFile string `json:"-"`
+	Credentials    string `json:"credentials,omitempty"`
+	StorageClass   string `json:"storage_class,omitempty"`
+}
+
+// GCSFs is a Fs implementation for Google Cloud Storage.
+type GCSFs struct {
+	connectionID   string
+	localTempDir   string
+	config         GCSFsConfig
+	svc            *storage.Client
+	ctxTimeout     time.Duration
+	ctxLongTimeout time.Duration
+}
+
+// NewGCSFs returns an GCSFs object that allows to interact with Google Cloud Storage
+func NewGCSFs(connectionID, localTempDir string, config GCSFsConfig) (Fs, error) {
+	var err error
+	fs := GCSFs{
+		connectionID:   connectionID,
+		localTempDir:   localTempDir,
+		config:         config,
+		ctxTimeout:     30 * time.Second,
+		ctxLongTimeout: 300 * time.Second,
+	}
+	if err = ValidateGCSFsConfig(&fs.config, fs.config.CredentialFile); err != nil {
+		return fs, err
+	}
+	ctx := context.Background()
+	fs.svc, err = storage.NewClient(ctx, option.WithCredentialsFile(fs.config.CredentialFile))
+	return fs, err
+}
+
+// Name returns the name for the Fs implementation
+func (fs GCSFs) Name() string {
+	return fmt.Sprintf("GCSFs bucket: %#v", fs.config.Bucket)
+}
+
+// ConnectionID returns the SSH connection ID associated to this Fs implementation
+func (fs GCSFs) ConnectionID() string {
+	return fs.connectionID
+}
+
+// Stat returns a FileInfo describing the named file
+func (fs GCSFs) Stat(name string) (os.FileInfo, error) {
+	var result FileInfo
+	var err error
+	if len(name) == 0 || name == "." {
+		err := fs.checkIfBucketExists()
+		if err != nil {
+			return result, err
+		}
+		return NewFileInfo(name, true, 0, time.Time{}), nil
+	}
+	if fs.config.KeyPrefix == name+"/" {
+		return NewFileInfo(name, true, 0, time.Time{}), nil
+	}
+	prefix := fs.getPrefixForStat(name)
+	query := &storage.Query{Prefix: prefix, Delimiter: "/"}
+	/*err = query.SetAttrSelection(gcsDefaultFieldsSelection)
+	if err != nil {
+		return result, err
+	}*/
+	ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
+	defer cancelFn()
+	bkt := fs.svc.Bucket(fs.config.Bucket)
+	it := bkt.Objects(ctx, query)
+	for {
+		attrs, err := it.Next()
+		if err == iterator.Done {
+			break
+		}
+		if err != nil {
+			metrics.GCSListObjectsCompleted(err)
+			return result, err
+		}
+		if len(attrs.Prefix) > 0 {
+			if fs.isEqual(attrs.Prefix, name) {
+				result = NewFileInfo(name, true, 0, time.Time{})
+			}
+		} else {
+			if !attrs.Deleted.IsZero() {
+				continue
+			}
+			if fs.isEqual(attrs.Name, name) {
+				isDir := strings.HasSuffix(attrs.Name, "/")
+				result = NewFileInfo(name, isDir, attrs.Size, attrs.Updated)
+			}
+		}
+	}
+	metrics.GCSListObjectsCompleted(nil)
+	if len(result.Name()) == 0 {
+		err = errors.New("404 no such file or directory")
+	}
+	return result, err
+}
+
+// Lstat returns a FileInfo describing the named file
+func (fs GCSFs) Lstat(name string) (os.FileInfo, error) {
+	return fs.Stat(name)
+}
+
+// Open opens the named file for reading
+func (fs GCSFs) Open(name string) (*os.File, *pipeat.PipeReaderAt, func(), error) {
+	r, w, err := pipeat.AsyncWriterPipeInDir(fs.localTempDir)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+	bkt := fs.svc.Bucket(fs.config.Bucket)
+	obj := bkt.Object(name)
+	ctx, cancelFn := context.WithCancel(context.Background())
+	objectReader, err := obj.NewReader(ctx)
+	if err != nil {
+		r.Close()
+		w.Close()
+		cancelFn()
+		return nil, nil, nil, err
+	}
+	go func() {
+		defer cancelFn()
+		defer objectReader.Close()
+		n, err := io.Copy(w, objectReader)
+		w.CloseWithError(err)
+		fsLog(fs, logger.LevelDebug, "download completed, path: %#v size: %v, err: %v", name, n, err)
+		metrics.GCSTransferCompleted(n, 1, err)
+	}()
+	return nil, r, cancelFn, nil
+}
+
+// Create creates or opens the named file for writing
+func (fs GCSFs) Create(name string, flag int) (*os.File, *pipeat.PipeWriterAt, func(), error) {
+	r, w, err := pipeat.PipeInDir(fs.localTempDir)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+	bkt := fs.svc.Bucket(fs.config.Bucket)
+	obj := bkt.Object(name)
+	ctx, cancelFn := context.WithCancel(context.Background())
+	objectWriter := obj.NewWriter(ctx)
+	if len(fs.config.StorageClass) > 0 {
+		objectWriter.ObjectAttrs.StorageClass = fs.config.StorageClass
+	}
+	go func() {
+		defer cancelFn()
+		defer objectWriter.Close()
+		n, err := io.Copy(objectWriter, r)
+		r.CloseWithError(err)
+		fsLog(fs, logger.LevelDebug, "upload completed, path: %#v, readed bytes: %v, err: %v", name, n, err)
+		metrics.GCSTransferCompleted(n, 0, err)
+	}()
+	return nil, w, cancelFn, nil
+}
+
+// Rename renames (moves) source to target.
+// We don't support renaming non empty directories since we should
+// rename all the contents too and this could take long time: think
+// about directories with thousands of files, for each file we should
+// execute a CopyObject call.
+func (fs GCSFs) Rename(source, target string) error {
+	if source == target {
+		return nil
+	}
+	fi, err := fs.Stat(source)
+	if err != nil {
+		return err
+	}
+	if fi.IsDir() {
+		contents, err := fs.ReadDir(source)
+		if err != nil {
+			return err
+		}
+		if len(contents) > 0 {
+			return fmt.Errorf("Cannot rename non empty directory: %#v", source)
+		}
+		if !strings.HasSuffix(source, "/") {
+			source += "/"
+		}
+		if !strings.HasSuffix(target, "/") {
+			target += "/"
+		}
+	}
+	src := fs.svc.Bucket(fs.config.Bucket).Object(source)
+	dst := fs.svc.Bucket(fs.config.Bucket).Object(target)
+	ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
+	defer cancelFn()
+	copier := dst.CopierFrom(src)
+	if len(fs.config.StorageClass) > 0 {
+		copier.StorageClass = fs.config.StorageClass
+	}
+	_, err = copier.Run(ctx)
+	metrics.GCSCopyObjectCompleted(err)
+	if err != nil {
+		return err
+	}
+	return fs.Remove(source, fi.IsDir())
+}
+
+// Remove removes the named file or (empty) directory.
+func (fs GCSFs) Remove(name string, isDir bool) error {
+	if isDir {
+		contents, err := fs.ReadDir(name)
+		if err != nil {
+			return err
+		}
+		if len(contents) > 0 {
+			return fmt.Errorf("Cannot remove non empty directory: %#v", name)
+		}
+		if !strings.HasSuffix(name, "/") {
+			name += "/"
+		}
+	}
+	ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
+	defer cancelFn()
+	err := fs.svc.Bucket(fs.config.Bucket).Object(name).Delete(ctx)
+	metrics.GCSDeleteObjectCompleted(err)
+	return err
+}
+
+// Mkdir creates a new directory with the specified name and default permissions
+func (fs GCSFs) Mkdir(name string) error {
+	_, err := fs.Stat(name)
+	if !fs.IsNotExist(err) {
+		return err
+	}
+	if !strings.HasSuffix(name, "/") {
+		name += "/"
+	}
+	_, w, _, err := fs.Create(name, 0)
+	if err != nil {
+		return err
+	}
+	return w.Close()
+}
+
+// Symlink creates source as a symbolic link to target.
+func (GCSFs) Symlink(source, target string) error {
+	return errors.New("403 symlinks are not supported")
+}
+
+// Chown changes the numeric uid and gid of the named file.
+// Silently ignored.
+func (GCSFs) Chown(name string, uid int, gid int) error {
+	return nil
+}
+
+// Chmod changes the mode of the named file to mode.
+// Silently ignored.
+func (GCSFs) Chmod(name string, mode os.FileMode) error {
+	return nil
+}
+
+// Chtimes changes the access and modification times of the named file.
+// Silently ignored.
+func (GCSFs) Chtimes(name string, atime, mtime time.Time) error {
+	return errors.New("403 chtimes is not supported")
+}
+
+// ReadDir reads the directory named by dirname and returns
+// a list of directory entries.
+func (fs GCSFs) ReadDir(dirname string) ([]os.FileInfo, error) {
+	var result []os.FileInfo
+	// dirname deve essere già cleaned
+	prefix := ""
+	if len(dirname) > 0 && dirname != "." {
+		prefix = strings.TrimPrefix(dirname, "/")
+		if !strings.HasSuffix(prefix, "/") {
+			prefix += "/"
+		}
+	}
+	query := &storage.Query{Prefix: prefix, Delimiter: "/"}
+	/*err := query.SetAttrSelection(gcsDefaultFieldsSelection)
+	if err != nil {
+		return result, err
+	}*/
+	ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
+	defer cancelFn()
+	bkt := fs.svc.Bucket(fs.config.Bucket)
+	it := bkt.Objects(ctx, query)
+	for {
+		attrs, err := it.Next()
+		if err == iterator.Done {
+			break
+		}
+		if err != nil {
+			metrics.GCSListObjectsCompleted(err)
+			return result, err
+		}
+		if len(attrs.Prefix) > 0 {
+			name, _ := fs.resolve(attrs.Prefix, prefix)
+			result = append(result, NewFileInfo(name, true, 0, time.Time{}))
+		} else {
+			name, isDir := fs.resolve(attrs.Name, prefix)
+			if len(name) == 0 {
+				continue
+			}
+			if !attrs.Deleted.IsZero() {
+				continue
+			}
+			result = append(result, NewFileInfo(name, isDir, attrs.Size, attrs.Updated))
+		}
+	}
+	metrics.GCSListObjectsCompleted(nil)
+	return result, nil
+}
+
+// IsUploadResumeSupported returns true if upload resume is supported.
+// SFTP Resume is not supported on S3
+func (GCSFs) IsUploadResumeSupported() bool {
+	return false
+}
+
+// IsAtomicUploadSupported returns true if atomic upload is supported.
+// S3 uploads are already atomic, we don't need to upload to a temporary
+// file
+func (GCSFs) IsAtomicUploadSupported() bool {
+	return false
+}
+
+// IsNotExist returns a boolean indicating whether the error is known to
+// report that a file or directory does not exist
+func (GCSFs) IsNotExist(err error) bool {
+	if err == nil {
+		return false
+	}
+	if err == storage.ErrObjectNotExist || err == storage.ErrBucketNotExist {
+		return true
+	}
+	if e, ok := err.(*googleapi.Error); ok {
+		if e.Code == http.StatusNotFound {
+			return true
+		}
+	}
+	return strings.Contains(err.Error(), "404")
+}
+
+// IsPermission returns a boolean indicating whether the error is known to
+// report that permission is denied.
+func (GCSFs) IsPermission(err error) bool {
+	if err == nil {
+		return false
+	}
+	if e, ok := err.(*googleapi.Error); ok {
+		if e.Code == http.StatusForbidden || e.Code == http.StatusUnauthorized {
+			return true
+		}
+	}
+	return strings.Contains(err.Error(), "403")
+}
+
+// CheckRootPath creates the specified root directory if it does not exists
+func (fs GCSFs) CheckRootPath(username string, uid int, gid int) bool {
+	// we need a local directory for temporary files
+	osFs := NewOsFs(fs.ConnectionID(), fs.localTempDir)
+	osFs.CheckRootPath(username, uid, gid)
+	return fs.checkIfBucketExists() != nil
+}
+
+// ScanRootDirContents returns the number of files contained in the bucket,
+// and their size
+func (fs GCSFs) ScanRootDirContents() (int, int64, error) {
+	numFiles := 0
+	size := int64(0)
+	query := &storage.Query{Prefix: fs.config.KeyPrefix}
+	err := query.SetAttrSelection(gcsDefaultFieldsSelection)
+	if err != nil {
+		return numFiles, size, err
+	}
+	ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxLongTimeout))
+	defer cancelFn()
+	bkt := fs.svc.Bucket(fs.config.Bucket)
+	it := bkt.Objects(ctx, query)
+	for {
+		attrs, err := it.Next()
+		if err == iterator.Done {
+			break
+		}
+		if err != nil {
+			metrics.GCSListObjectsCompleted(err)
+			return numFiles, size, err
+		}
+		if !attrs.Deleted.IsZero() {
+			continue
+		}
+		numFiles++
+		size += attrs.Size
+	}
+	metrics.GCSListObjectsCompleted(nil)
+	return numFiles, size, err
+}
+
+// GetAtomicUploadPath returns the path to use for an atomic upload.
+// S3 uploads are already atomic, we never call this method for S3
+func (GCSFs) GetAtomicUploadPath(name string) string {
+	return ""
+}
+
+// GetRelativePath returns the path for a file relative to the user's home dir.
+// This is the path as seen by SFTP users
+func (fs GCSFs) GetRelativePath(name string) string {
+	rel := path.Clean(name)
+	if rel == "." {
+		rel = ""
+	}
+	if !path.IsAbs(rel) {
+		rel = "/" + rel
+	}
+	if len(fs.config.KeyPrefix) > 0 {
+		if !strings.HasPrefix(rel, "/"+fs.config.KeyPrefix) {
+			rel = "/"
+		}
+		rel = path.Clean("/" + strings.TrimPrefix(rel, "/"+fs.config.KeyPrefix))
+	}
+	return rel
+}
+
+// Join joins any number of path elements into a single path
+func (GCSFs) Join(elem ...string) string {
+	return strings.TrimPrefix(path.Join(elem...), "/")
+}
+
+// ResolvePath returns the matching filesystem path for the specified sftp path
+func (fs GCSFs) ResolvePath(sftpPath string) (string, error) {
+	if !path.IsAbs(sftpPath) {
+		sftpPath = path.Clean("/" + sftpPath)
+	}
+	return fs.Join(fs.config.KeyPrefix, strings.TrimPrefix(sftpPath, "/")), nil
+}
+
+func (fs *GCSFs) resolve(name string, prefix string) (string, bool) {
+	result := strings.TrimPrefix(name, prefix)
+	isDir := strings.HasSuffix(result, "/")
+	if isDir {
+		result = strings.TrimSuffix(result, "/")
+	}
+	return result, isDir
+}
+
+func (fs *GCSFs) isEqual(key string, sftpName string) bool {
+	if key == sftpName {
+		return true
+	}
+	if key == sftpName+"/" {
+		return true
+	}
+	if key+"/" == sftpName {
+		return true
+	}
+	return false
+}
+
+func (fs *GCSFs) checkIfBucketExists() error {
+	ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
+	defer cancelFn()
+	bkt := fs.svc.Bucket(fs.config.Bucket)
+	_, err := bkt.Attrs(ctx)
+	metrics.GCSHeadBucketCompleted(err)
+	return err
+}
+
+func (fs *GCSFs) getPrefixForStat(name string) string {
+	prefix := path.Dir(name)
+	if prefix == "/" || prefix == "." || len(prefix) == 0 {
+		prefix = ""
+	} else {
+		prefix = strings.TrimPrefix(prefix, "/")
+		if !strings.HasSuffix(prefix, "/") {
+			prefix += "/"
+		}
+	}
+	return prefix
+}

vfs/s3fs.go

@@ -21,15 +21,15 @@ import (
 	"github.com/eikenb/pipeat"
 )
 
-// S3FsConfig defines the configuration for S3fs
+// S3FsConfig defines the configuration for S3 based filesystem
 type S3FsConfig struct {
 	Bucket string `json:"bucket,omitempty"`
 	// KeyPrefix is similar to a chroot directory for local filesystem.
-	// If specified the SFTP user will only see contents that starts with
+	// If specified the SFTP user will only see objects that starts with
 	// this prefix and so you can restrict access to a specific virtual
 	// folder. The prefix, if not empty, must not start with "/" and must
 	// end with "/".
-	//If empty the whole bucket contents will be available
+	// If empty the whole bucket contents will be available
 	KeyPrefix    string `json:"key_prefix,omitempty"`
 	Region       string `json:"region,omitempty"`
 	AccessKey    string `json:"access_key,omitempty"`
@@ -70,7 +70,6 @@ func NewS3Fs(connectionID, localTempDir string, config S3FsConfig) (Fs, error) {
 		Region:      aws.String(fs.config.Region),
 		Credentials: credentials.NewStaticCredentials(fs.config.AccessKey, fs.config.AccessSecret, ""),
 	}
-	//config.WithLogLevel(aws.LogDebugWithHTTPBody)
 	if len(fs.config.Endpoint) > 0 {
 		awsConfig.Endpoint = aws.String(fs.config.Endpoint)
 		awsConfig.S3ForcePathStyle = aws.Bool(true)
@@ -95,16 +94,16 @@ func (fs S3Fs) ConnectionID() string {
 
 // Stat returns a FileInfo describing the named file
 func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
-	var result S3FileInfo
+	var result FileInfo
 	if name == "/" || name == "." {
 		err := fs.checkIfBucketExists()
 		if err != nil {
 			return result, err
 		}
-		return NewS3FileInfo(name, true, 0, time.Time{}), nil
+		return NewFileInfo(name, true, 0, time.Time{}), nil
 	}
 	if "/"+fs.config.KeyPrefix == name+"/" {
-		return NewS3FileInfo(name, true, 0, time.Time{}), nil
+		return NewFileInfo(name, true, 0, time.Time{}), nil
 	}
 	prefix := path.Dir(name)
 	if prefix == "/" || prefix == "." {
@@ -124,7 +123,7 @@ func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
 	}, func(page *s3.ListObjectsV2Output, lastPage bool) bool {
 		for _, p := range page.CommonPrefixes {
 			if fs.isEqual(p.Prefix, name) {
-				result = NewS3FileInfo(name, true, 0, time.Time{})
+				result = NewFileInfo(name, true, 0, time.Time{})
 				return false
 			}
 		}
@@ -133,7 +132,7 @@ func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
 				objectSize := *fileObject.Size
 				objectModTime := *fileObject.LastModified
 				isDir := strings.HasSuffix(*fileObject.Key, "/")
-				result = NewS3FileInfo(name, isDir, objectSize, objectModTime)
+				result = NewFileInfo(name, isDir, objectSize, objectModTime)
 				return false
 			}
 		}
@@ -325,7 +324,7 @@ func (fs S3Fs) ReadDir(dirname string) ([]os.FileInfo, error) {
 	}, func(page *s3.ListObjectsV2Output, lastPage bool) bool {
 		for _, p := range page.CommonPrefixes {
 			name, isDir := fs.resolve(p.Prefix, prefix)
-			result = append(result, NewS3FileInfo(name, isDir, 0, time.Time{}))
+			result = append(result, NewFileInfo(name, isDir, 0, time.Time{}))
 		}
 		for _, fileObject := range page.Contents {
 			objectSize := *fileObject.Size
@@ -334,7 +333,7 @@ func (fs S3Fs) ReadDir(dirname string) ([]os.FileInfo, error) {
 			if len(name) == 0 {
 				continue
 			}
-			result = append(result, NewS3FileInfo(name, isDir, objectSize, objectModTime))
+			result = append(result, NewFileInfo(name, isDir, objectSize, objectModTime))
 		}
 		return true
 	})
@@ -394,23 +393,7 @@ func (fs S3Fs) CheckRootPath(username string, uid int, gid int) bool {
 	// we need a local directory for temporary files
 	osFs := NewOsFs(fs.ConnectionID(), fs.localTempDir)
 	osFs.CheckRootPath(username, uid, gid)
-	err := fs.checkIfBucketExists()
-	if err == nil {
-		return true
-	}
-	if !fs.IsNotExist(err) {
-		return false
-	}
-	ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
-	defer cancelFn()
-	input := &s3.CreateBucketInput{
-		Bucket: aws.String(fs.config.Bucket),
-	}
-	_, err = fs.svc.CreateBucketWithContext(ctx, input)
-	fsLog(fs, logger.LevelDebug, "bucket %#v for user %#v does not exists, try to create, error: %v",
-		fs.config.Bucket, username, err)
-	metrics.S3CreateBucketCompleted(err)
-	return err == nil
+	return fs.checkIfBucketExists() != nil
 }
 
 // ScanRootDirContents returns the number of files contained in the bucket,

vfs/vfs.go

@@ -3,6 +3,7 @@ package vfs
 
 import (
 	"errors"
+	"fmt"
 	"os"
 	"path"
 	"runtime"
@@ -14,7 +15,7 @@ import (
 	"github.com/pkg/sftp"
 )
 
-// Fs defines the interface for filesystems backends
+// Fs defines the interface for filesystem backends
 type Fs interface {
 	Name() string
 	ConnectionID() string
@@ -94,6 +95,32 @@ func ValidateS3FsConfig(config *S3FsConfig) error {
 	return nil
 }
 
+// ValidateGCSFsConfig returns nil if the specified GCS config is valid, otherwise an error
+func ValidateGCSFsConfig(config *GCSFsConfig, credentialsFilePath string) error {
+	if len(config.Bucket) == 0 {
+		return errors.New("bucket cannot be empty")
+	}
+	if len(config.KeyPrefix) > 0 {
+		if strings.HasPrefix(config.KeyPrefix, "/") {
+			return errors.New("key_prefix cannot start with /")
+		}
+		config.KeyPrefix = path.Clean(config.KeyPrefix)
+		if !strings.HasSuffix(config.KeyPrefix, "/") {
+			config.KeyPrefix += "/"
+		}
+	}
+	if len(config.Credentials) == 0 {
+		fi, err := os.Stat(credentialsFilePath)
+		if err != nil {
+			return fmt.Errorf("invalid credentials %v", err)
+		}
+		if fi.Size() == 0 {
+			return errors.New("credentials cannot be empty")
+		}
+	}
+	return nil
+}
+
 // SetPathPermissions calls fs.Chown.
 // It does nothing for local filesystem on windows
 func SetPathPermissions(fs Fs, path string, uid int, gid int) {