Merge pull request #724 from logseq/feat/chrome-native-fs

Chrome native file system api integration

README.md

@@ -62,51 +62,12 @@ The following is for developers and designers who want to build and run Logseq l
 
 ## Set up development environment
 
-If you are on Windows, use the [Windows setup](#windows-setup) below.
-
 ### 1. Requirements
 
-- [Java & Clojure](https://clojure.org/guides/getting_started)
-
-- [PostgreSQL](https://www.postgresql.org/download/)
-
 - [Node.js](https://nodejs.org/en/download/) & [Yarn](https://classic.yarnpkg.com/en/docs/install/)
+- [Java & Clojure](https://clojure.org/guides/getting_started)
 
-### 2. Create a GitHub app
-
-Follow the guide at <https://docs.github.com/en/free-pro-team@latest/developers/apps/creating-a-github-app>, where the user authorization "Callback URL" should be `http://localhost:3000/auth/github`.
-
-Remember to download the `private-key.pem` which will be used for the next step. Also take note of your `App ID`, `Client ID`, and your newly generated `Client Secret` for use in step 4.
-
-![Screenshot 2020-11-27 22-22-39 +0800](https://user-images.githubusercontent.com/479169/100460276-e0bad100-3101-11eb-8fed-1f7c85824b62.png)
-
-**Add contents permission**:
-![Screenshot 2020-11-27 22-22-57 +0800](https://user-images.githubusercontent.com/479169/100460271-def10d80-3101-11eb-91bb-f2339a52d4f8.png)
-
-
-### 3. Set up PostgreSQL
-
-Make sure you have PostgreSQL running. You can check if it's running with `pg_ctl -D /usr/local/var/postgres status` and use `pg_ctl -D /usr/local/var/postgres start` to start it up. You'll also need to make a Logseq DB in PostgreSQL. Do that with `createdb logseq`.
-
-### 4. Add environment variables
-
-``` bash
-export ENVIRONMENT="dev"
-export JWT_SECRET="xxxxxxxxxxxxxxxxxxxx"
-export COOKIE_SECRET="xxxxxxxxxxxxxxxxxxxx"
-export DATABASE_URL="postgres://localhost:5432/logseq"
-export GITHUB_APP2_NAME="logseq-test-your-username-app"
-export GITHUB_APP2_ID="your id"
-export GITHUB_APP2_KEY="xxxxxxxxxxxxxxxxxxxx" #Your Github App's Client ID
-export GITHUB_APP2_SECRET="xxxxxxxxxxxxxxxxxxxx"
-# Replace your-code-directory and your-app.private-key.pem with yours
-export GITHUB_APP_PEM="/your-code-directory/your-app.private-key.pem"
-export LOG_PATH="/tmp/logseq"
-export PG_USERNAME="xxx"
-export PG_PASSWORD="xxx"
-```
-
-### 5. Compile to JavaScript
+### 2. Compile to JavaScript
 
 ``` bash
 git clone https://github.com/logseq/logseq
@@ -114,46 +75,12 @@ yarn
 yarn watch
 ```
 
-### 6. Start the Clojure server
-
-1.  Download jar
+### 3. Open the browser
 
-    Go to <https://github.com/logseq/logseq/releases>, download the `logseq.jar` and put it in the `logseq` directory.
+Open <http://localhost:3001>.
 
-2.  Run jar
+### 4. Build a release
 
-    ``` bash
-    java -Duser.timezone=UTC -jar logseq.jar
-    ```
-
-### 7. Open the browser
-
-Open <http://localhost:3000>.
-
-## Windows setup
-
-### 1. Required software
-
-Install Clojure through scoop-clojure: <https://github.com/littleli/scoop-clojure>. You can also install [Node.js](https://nodejs.org/en/), [Yarn](https://yarnpkg.com/) and [PostgreSQL](https://www.postgresql.org/download/) through scoop if you want to.
-
-### 2. Create a GitHub app
-
-Follow [Step 2](#2-create-a-github-app) above if you want Logseq to connect to GitHub. If not, skip this section. The `GITHUB_APP_PEM` variable in the `run-windows.bat` needs to be set with the correct directory for your system.
-
-### 3. Set up PostgreSQL
-
-Make sure you have PostgreSQL running. You can check if it's running with `pg_ctl status` and use `pg_ctl start` to start it up. You'll also need to make a Logseq DB in PostgreSQL. Do that with `createdb logseq`.
-
-### 4. Download the Clojure server
-
-Go to <https://github.com/logseq/logseq/releases>, download the `logseq.jar` and move into the root directory of repo.
-
-### 5. Start Logseq
-
-Run `start-windows.bat` which is located in the repo. This will open a second terminal that runs Logseq's backend server. To completely stop Logseq, you'll need to also close that second terminal that was opened.
-
-`start-windows.bat` will try to start PostgreSQL for you if it's not already started.
-
-## Build errors
-### 1. The required namespace `devtools.preload` is not available.
-Upload your clojure to at least version `1.10.1.739`.
+``` bash
+yarn release
+```

clojure.bat

@@ -1,2 +0,0 @@
-@echo off
-cmd-clojure %*

package.json

@@ -4,6 +4,7 @@
     "private": true,
     "devDependencies": {
         "@tailwindcss/ui": "^0.1.3",
+        "cross-env": "^7.0.3",
         "cssnano": "^4.1.10",
         "del": "^6.0.0",
         "gulp": "^4.0.2",
@@ -18,7 +19,7 @@
         "tailwindcss": "^1.3.4"
     },
     "scripts": {
-        "watch": "run-p cljs:watch gulp:watch",
+        "watch": "run-p cljs:watch gulp:build gulp:watch",
         "release": "run-s cljs:release gulp:build",
         "watch-app": "run-p cljs:watch-app gulp:watch",
         "release-app": "run-s cljs:release-app gulp:build",
@@ -26,8 +27,9 @@
         "dev-release-app": "run-s cljs:dev-release-app gulp:build",
         "clean": "gulp clean",
         "test": "run-s cljs:test cljs:run-test",
+        "report": "run-s cljs:report",
         "gulp:watch": "gulp watch",
-        "gulp:build": "NODE_ENV=production gulp build",
+        "gulp:build": "cross-env NODE_ENV=production gulp build",
         "cljs:watch": "clojure -M:cljs watch app publishing",
         "cljs:release": "clojure -M:cljs release app publishing",
         "cljs:test": "clojure -A:test compile test",
@@ -43,6 +45,7 @@
         "codemirror": "^5.58.1",
         "diff": "^4.0.2",
         "dropbox": "^5.2.0",
+        "ignore": "^5.1.8",
         "jszip": "^3.5.0",
         "localforage": "^1.7.3",
         "mousetrap": "^1.6.5",

public/static

@@ -0,0 +1 @@
+../static

resources/css/common.css

@@ -444,6 +444,10 @@ svg {
   }
 }
 
+.loader {
+    -webkit-animation: lds-dual-ring 2s infinite linear;
+}
+
 .canceled {
   text-decoration: line-through;
 }
@@ -1121,23 +1125,13 @@ pre {
   margin: 1em 0 0.5em;
 }
 
-
-/* .page .content *, #intro .content *, .page .title { */
-/*     margin-left: auto; */
-/*     margin-right: auto; */
-/* } */
-
-/* .page .blocks-container .ls-block { */
-/*     align-self: center; */
-/* } */
-
 #intro .content {
   flex-direction: column;
   align-items: center;
 }
 
-#intro .content :not(img), .foldable-title {
-  max-width: 665px;
+.ls-block, .foldable-title {
+    max-width: 700px;
 }
 
 .ls-block, .editor-wrapper {
@@ -1153,10 +1147,6 @@ img, video {
   margin-right: auto;
 }
 
-#intro .intro-docs, img, video, .intro .ls-block {
-  max-width: 653px;
-}
-
 .ls-block img {
   box-shadow: 0 20px 25px -5px rgba(0, 0, 0, .1), 0 10px 10px -5px rgba(0, 0, 0, .04);
 }

resources/index.html

@@ -1,24 +0,0 @@
-<!DOCTYPE html>
-<html><head><meta charset="utf-8"><meta content="minimum-scale=1, initial-scale=1, width=device-width, shrink-to-fit=no" name="viewport"><meta content="Agp2znmEoRKqxMhzbNL2R3UOCNcagP7+fu0KSM+09O21u7EHdJgqhTrslpfyFC/dSt6jvpaDzNiFf2769fLHMAUAAABoeyJvcmlnaW4iOiJodHRwczovL2xvZ3NlcS5jb206NDQzIiwiZmVhdHVyZSI6Ik5hdGl2ZUZpbGVTeXN0ZW0yIiwiZXhwaXJ5IjoxNTk3Mjg5MzY5LCJpc1N1YmRvbWFpbiI6dHJ1ZX0=" http-equiv="origin-trial"><link href="https://asset.logseq.com/static/style.css" rel="stylesheet" type="text/css"><link href="https://asset.logseq.com/static/img/logo.png" rel="shortcut icon" type="image/png"><link href="https://asset.logseq.com/static/img/logo.png" rel="shortcut icon" sizes="192x192"><link href="https://asset.logseq.com/static/img/logo.png" rel="apple-touch-icon"><meta content="summary" name="twitter:card"><meta content="A local-first notes app which uses Git to store and sync your knowledge." name="twitter:description"><meta content="@logseq" name="twitter:site"><meta content="A local-first notes app." name="twitter:title"><meta content="https://asset.logseq.com/static/img/logo.png" name="twitter:image:src"><meta content="A local-first notes app." name="twitter:image:alt"><meta content="A local-first notes app." property="og:title"><meta content="site" property="og:type"><meta content="https://logseq.com" property="og:url"><meta content="https://asset.logseq.com/static/img/logo.png" property="og:image"><meta content="A local-first notes app which uses Git to store and sync your knowledge." property="og:description"><title>Logseq: A local-first notes app</title><meta content="logseq" property="og:site_name"><meta description="A local-first notes app which uses Git to store and sync your knowledge."><script crossorigin="anonymous" defer onload="if (window.location.host != &apos;localhost:3000&apos;) {
-          Sentry.init({dsn: &apos;https://[email protected]/5311485&apos;});
-};" src="https://asset.logseq.com/static/js/sentry.min.js"></script></head><body><div id="root"></div><script>window.user={"name":"tiensonqin","email":"[email protected]","avatar":"https://avatars3.githubusercontent.com/u/479169?v=4","repos":[{"id":"bc80efff-1420-4eb7-9e07-9506b8d9bbe0","url":"https://github.com/tiensonqin/notes"}],"preferred_format":"org","encrypt_object_key":"snRsaP8r9VG6KsXxu0IfDA"};</script><script src="https://asset.logseq.com/static/js/mldoc.min.js"></script><script src="/js/magic_portal.js"></script><script>let worker = new Worker("/js/worker.js");
-const portal = new MagicPortal(worker);
-;(async () => {
-  const git = await portal.get('git');
-  window.git = git;
-  const fs = await portal.get('fs');
-  window.fs = fs;
-  const pfs = await portal.get('pfs');
-  window.pfs = pfs;
-  const workerThread = await portal.get('workerThread');
-  window.workerThread = workerThread;
-})();
-</script><script src="https://asset.logseq.com/static/js/main.js"></script><script>
-  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
-  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
-  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
-  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-
-  ga('create', 'UA-171599883-1', 'logseq.com');
-  ga('send', 'pageview');
-</script></body></html>

resources/js/worker.js

@@ -0,0 +1,303 @@
+importScripts(
+  // Batched optimization
+  "/static/js/lightning-fs.min.js?v=0.0.2.3",
+  "https://cdn.jsdelivr.net/npm/[email protected]/index.umd.min.js",
+  "https://cdn.jsdelivr.net/npm/[email protected]/http/web/index.umd.js",
+  // Fixed a bug
+  "/static/js/magic_portal.js"
+);
+
+const detect = () => {
+  if (typeof window !== 'undefined' && !self.skipWaiting) {
+    return 'window'
+  } else if (typeof self !== 'undefined' && !self.skipWaiting) {
+    return 'Worker'
+  } else if (typeof self !== 'undefined' && self.skipWaiting) {
+    return 'ServiceWorker'
+  }
+};
+
+function basicAuth (username, token) {
+  return "Basic " + btoa(username + ":" + token);
+}
+
+const fsName = 'logseq';
+const createFS = () => new LightningFS(fsName);
+let fs = createFS();
+let pfs = fs.promises;
+
+if (detect() === 'Worker') {
+  const portal = new MagicPortal(self);
+  portal.set('git', git);
+  portal.set('fs', fs);
+  portal.set('pfs', pfs);
+  portal.set('gitHttp', GitHttp);
+  portal.set('workerThread', {
+    setConfig: function (dir, path, value) {
+      return git.setConfig ({
+        fs,
+        dir,
+        path,
+        value
+      });
+    },
+    clone: function (dir, url, corsProxy, depth, branch, username, token) {
+      return git.clone ({
+        fs,
+        dir,
+        http: GitHttp,
+        url,
+        corsProxy,
+        ref: branch,
+        singleBranch: true,
+        depth,
+        headers: {
+          "Authorization": basicAuth(username, token)
+        }
+      });
+    },
+    fetch: function (dir, url, corsProxy, depth, branch, username, token) {
+      return git.fetch ({
+        fs,
+        dir,
+        http: GitHttp,
+        url,
+        corsProxy,
+        ref: branch,
+        singleBranch: true,
+        depth,
+        headers: {
+          "Authorization": basicAuth(username, token)
+        }
+      });
+    },
+    pull: function (dir, corsProxy, branch, username, token) {
+      return git.pull ({
+        fs,
+        dir,
+        http: GitHttp,
+        corsProxy,
+        ref: branch,
+        singleBranch: true,
+        // fast: true,
+        headers: {
+          "Authorization": basicAuth(username, token)
+        }
+      });
+    },
+    push: function (dir, corsProxy, branch, force, username, token) {
+      return git.push ({
+        fs,
+        dir,
+        http: GitHttp,
+        ref: branch,
+        corsProxy,
+        remote: "origin",
+        force,
+        headers: {
+          "Authorization": basicAuth(username, token)
+        }
+      });
+    },
+    merge: function (dir, branch) {
+      return git.merge ({
+        fs,
+        dir,
+        ours: branch,
+        theirs: "remotes/origin/" + branch,
+        // fastForwardOnly: true
+      });
+    },
+    checkout: function (dir, branch) {
+      return git.checkout ({
+        fs,
+        dir,
+        ref: branch,
+      });
+    },
+    log: function (dir, branch, depth) {
+      return git.log ({
+        fs,
+        dir,
+        ref: branch,
+        depth,
+        singleBranch: true
+      })
+    },
+    add: function (dir, file) {
+      return git.add ({
+        fs,
+        dir,
+        filepath: file
+      });
+    },
+    remove: function (dir, file) {
+      return git.remove ({
+        fs,
+        dir,
+        filepath: file
+      });
+    },
+    commit: function (dir, message, name, email, parent) {
+      if (parent) {
+        return git.commit ({
+          fs,
+          dir,
+          message,
+          author: {name: name,
+                   email: email},
+          parent: parent
+        });
+      } else {
+        return git.commit ({
+          fs,
+          dir,
+          message,
+          author: {name: name,
+                   email: email}
+        });
+      }
+    },
+    readCommit: function (dir, oid) {
+      return git.readCommit ({
+        fs,
+        dir,
+        oid
+      });
+    },
+    readBlob: function (dir, oid, path) {
+      return git.readBlob ({
+        fs,
+        dir,
+        oid,
+        path
+      });
+    },
+    writeRef: function (dir, branch, oid) {
+      return git.writeRef ({
+        fs,
+        dir,
+        ref: "refs/heads/" + branch,
+        value: oid,
+        force: true
+      });
+    },
+    resolveRef: function (dir, ref) {
+      return git.resolveRef ({
+        fs,
+        dir,
+        ref
+      });
+    },
+    listFiles: function (dir, branch) {
+      return git.listFiles ({
+        fs,
+        dir,
+        ref: branch
+      });
+    },
+    rimraf: async function (path) {
+      // try {
+      //   // First assume path is itself a file
+      //   await pfs.unlink(path)
+      //   // if that worked we're done
+      //   return
+      // } catch (err) {
+      //   // Otherwise, path must be a directory
+      //   if (err.code !== 'EISDIR') throw err
+      // }
+      // Knowing path is a directory,
+      // first, assume everything inside path is a file.
+      let files = await pfs.readdir(path);
+      for (let file of files) {
+        let child = path + '/' + file
+        try {
+          await pfs.unlink(child)
+        } catch (err) {
+          if (err.code !== 'EISDIR') throw err
+        }
+      }
+      // Assume what's left are directories and recurse.
+      let dirs = await pfs.readdir(path)
+      for (let dir of dirs) {
+        let child = path + '/' + dir
+        await rimraf(child, pfs)
+      }
+      // Finally, delete the empty directory
+      await pfs.rmdir(path)
+    },
+    getFileStateChanges: async function (commitHash1, commitHash2, dir) {
+      return git.walk({
+        fs,
+        dir,
+        trees: [git.TREE({ ref: commitHash1 }), git.TREE({ ref: commitHash2 })],
+        map: async function(filepath, [A, B]) {
+          var type = 'equal';
+          if (A === null) {
+            type = "add";
+          }
+
+          if (B === null) {
+            type = "remove";
+          }
+
+          // ignore directories
+          if (filepath === '.') {
+            return
+          }
+          if ((A !== null && (await A.type()) === 'tree')
+              ||
+              (B !== null && (await B.type()) === 'tree')) {
+            return
+          }
+
+          // generate ids
+          const Aoid = A !== null && await A.oid();
+          const Boid = B !== null && await B.oid();
+
+          if (type === "equal") {
+            // determine modification type
+            if (Aoid !== Boid) {
+              type = 'modify'
+            }
+            if (Aoid === undefined) {
+              type = 'add'
+            }
+            if (Boid === undefined) {
+              type = 'remove'
+            }
+          }
+
+          if (Aoid === undefined && Boid === undefined) {
+            console.log('Something weird happened:')
+            console.log(A)
+            console.log(B)
+          }
+
+          return {
+            path: `/${filepath}`,
+            type: type,
+          }
+        },
+      })
+    },
+    statusMatrix: async function (dir) {
+      await git.statusMatrix({ fs, dir });
+    },
+    getChangedFiles: async function (dir) {
+      try {
+        const FILE = 0, HEAD = 1, WORKDIR = 2;
+
+        let filenames = (await git.statusMatrix({ fs, dir }))
+            .filter(row => row[HEAD] !== row[WORKDIR])
+            .map(row => row[FILE]);
+
+        return filenames;
+      } catch (err) {
+        console.error(err);
+        return [];
+      }
+    }
+  });
+  // self.addEventListener("message", ({ data }) => console.log(data));
+}

shadow-cljs.edn

@@ -26,6 +26,8 @@
    {:before-load frontend.core/stop
     ;; after live-reloading finishes call this function
     :after-load frontend.core/start
+    :http-root    "public"
+    :http-port    3001
     :preloads     [devtools.preload]}}
 
   :test

src/main/frontend/components/block.cljs

@@ -1093,8 +1093,7 @@
      [:div.flex-1.flex-col.relative.block-content
       (cond-> {:id (str "block-content-" uuid)
                :style {:cursor "text"
-                       :min-height 24
-                       :max-width 560}}
+                       :min-height 24}}
         (not slide?)
         (merge attrs))
 
@@ -1502,9 +1501,9 @@
 
 (rum/defcs custom-query < rum/reactive
   {:will-mount (fn [state]
-                 (let [[config query] (:rum/args state)]
-                   (let [query-atom (db/custom-query query)]
-                     (assoc state :query-atom query-atom))))
+                 (let [[config query] (:rum/args state)
+                       query-atom (db/custom-query query)]
+                   (assoc state :query-atom query-atom)))
    :did-mount (fn [state]
                 (when-let [query (last (:rum/args state))]
                   (state/add-custom-query-component! query (:rum/react-component state)))
@@ -1552,11 +1551,13 @@
                                :margin-left "0.25rem"}})
 
             (seq result)                     ;TODO: table
-            [:pre
-             (for [record result]
-               (if (map? record)
-                 (str (util/pp-str record) "\n")
-                 record))]
+            (let [result (->>
+                          (for [record result]
+                            (if (map? record)
+                              (str (util/pp-str record) "\n")
+                              record))
+                          (remove nil?))]
+              [:pre result])
 
             :else
             [:div.text-sm.mt-2.ml-2.font-medium.opacity-50 "Empty"])

src/main/frontend/components/editor.cljs

@@ -698,6 +698,10 @@
                           current-pos (:pos (util/get-caret-pos (gdom/getElement id)))]
                       (state/set-edit-content! id value)
                       (state/set-edit-pos! current-pos)
+                      (when-let [repo (or (:block/repo block)
+                                          (state/get-current-repo))]
+                        (state/set-editor-last-input-time! repo (util/time-ms))
+                        (db/clear-repo-persistent-job! repo))
                       (let [input (gdom/getElement id)
                             native-e (gobj/get e "nativeEvent")
                             last-input-char (util/nth-safe value (dec current-pos))]

src/main/frontend/components/header.cljs

@@ -14,7 +14,8 @@
             [frontend.components.svg :as svg]
             [frontend.components.repo :as repo]
             [frontend.components.page :as page]
-            [frontend.components.search :as search]))
+            [frontend.components.search :as search]
+            [frontend.handler.web.nfs :as nfs]))
 
 (rum/defc logo < rum/reactive
   [{:keys [white?]}]
@@ -57,13 +58,8 @@
          {:title (t :graph)
           :options {:href (rfe/href :graph)}
           :icon svg/graph-sm})
-       (when (and logged? current-repo)
-         {:title (t :publishing)
-          :options {:on-click (fn []
-                                (export/export-repo-as-html! current-repo))}
-          :icon nil})
-       (when logged?
-         {:title (t :all-repos)
+       (when (or logged? (and (nfs/supported?) current-repo))
+         {:title (t :all-graphs)
           :options {:href (rfe/href :repos)}
           :icon svg/repos-sm})
        (when current-repo
@@ -78,12 +74,20 @@
          {:title (t :all-journals)
           :options {:href (rfe/href :all-journals)}
           :icon svg/calendar-sm})
-       {:title (t :excalidraw-title)
-        :options {:href (rfe/href :draw)}
-        :icon (svg/excalidraw-logo)}
        {:title (t :settings)
         :options {:href (rfe/href :settings)}
         :icon svg/settings-sm}
+       (when-let [project (and current-repo (state/get-current-project))]
+         (let [link (str config/website "/" project)]
+           {:title (str (t :go-to) "/" project)
+            :options {:href link
+                      :target "_blank"}
+            :icon svg/external-link}))
+       (when (and logged? current-repo)
+         {:title (t :export)
+          :options {:on-click (fn []
+                                (export/export-repo-as-html! current-repo))}
+          :icon nil})
        (when current-repo
          {:title (t :import)
           :options {:href (rfe/href :import)}
@@ -109,56 +113,59 @@
 
 (rum/defc header
   [{:keys [open-fn current-repo white? logged? page? route-match me default-home new-block-mode]}]
-  (rum/with-context [[t] i18n/*tongue-context*]
-    [:div.cp__header#head
-     (left-menu-button {:on-click (fn []
-                               (open-fn)
-                               (state/set-left-sidebar-open! true))})
-
-     (logo {:white? white?})
-
-     (if current-repo
-       (search/search)
-       [:div.flex-1])
-
-     (new-block-mode)
-
-     (when (and (not logged?)
-                (not config/publishing?))
-       [:a.text-sm.font-medium.login.opacity-70.hover:opacity-100
-        {:href "/login/github"
-         :on-click (fn []
-                     (storage/remove :git/current-repo))}
-        (t :login-github)])
-
-     (repo/sync-status)
-
-     [:div.repos.hidden.md:block
-      (repo/repos-dropdown true)]
-
-     (when-let [project (and current-repo (state/get-current-project))]
-       [:a.opacity-70.hover:opacity-100.ml-4
-        {:title (str (t :go-to) "/" project)
-         :href (str config/website "/" project)
-         :target "_blank"}
-        svg/external-link])
-
-     (when (and page? current-repo (not config/mobile?))
-       (let [page (get-in route-match [:path-params :name])
-             page (string/lower-case (util/url-decode page))
-             page (db/entity [:page/name page])]
-         (page/presentation current-repo page (:journal? page))))
-
-     (if config/publishing?
-       [:a.text-sm.font-medium.ml-3 {:href (rfe/href :graph)}
-        (t :graph)]
-
-       (dropdown-menu {:me me
-                       :t t
-                       :current-repo current-repo
-                       :default-home default-home}))
-
-     [:a#download-as-html.hidden]
-     [:a#download-as-zip.hidden]
-
-     (right-menu-button)]))
+  (let [local-repo? (= current-repo config/local-repo)
+        repos (->> (state/sub [:me :repos])
+                   (remove #(= (:url %) config/local-repo)))]
+    (rum/with-context [[t] i18n/*tongue-context*]
+      [:div.cp__header#head
+       (left-menu-button {:on-click (fn []
+                                      (open-fn)
+                                      (state/set-left-sidebar-open! true))})
+
+       (logo {:white? white?})
+
+       (if current-repo
+         (search/search)
+         [:div.flex-1])
+
+       (new-block-mode)
+
+       (when (and (not logged?)
+                  (not config/publishing?))
+         [:a.text-sm.font-medium.login.opacity-70.hover:opacity-100
+          {:href "/login/github"
+           :on-click (fn []
+                       (storage/remove :git/current-repo))}
+          (t :login-github)])
+
+       (repo/sync-status)
+
+       [:div.repos.hidden.md:block
+        (repo/repos-dropdown true)]
+
+       (when (and (nfs/supported?) (empty? repos))
+         (ui/tooltip
+          "Warning: this is an experimental feature, please only use it for testing purpose."
+          [:a.text-sm.font-medium.opacity-70.hover:opacity-100.ml-3.block
+           {:on-click (fn []
+                        (nfs/ls-dir-files))}
+           [:div.flex.flex-row.text-center
+            [:span.inline-block svg/folder-add]
+            (when-not config/mobile?
+              [:span.ml-1 {:style {:margin-top 2}}
+               (t :open)])]]
+          {:label-style {:width 200}}))
+
+       (if config/publishing?
+         [:a.text-sm.font-medium.ml-3 {:href (rfe/href :graph)}
+          (t :graph)]
+
+         (dropdown-menu {:me me
+                         :t t
+                         :current-repo current-repo
+                         :default-home default-home}))
+
+       [:a#download-as-html.hidden]
+       [:a#download-as-zip.hidden]
+
+       (right-menu-button)])))

src/main/frontend/components/journal.cljs

@@ -10,6 +10,7 @@
             [frontend.db :as db]
             [frontend.state :as state]
             [frontend.ui :as ui]
+            [frontend.config :as config]
             [frontend.components.content :as content]
             [frontend.components.block :as block]
             [frontend.components.editor :as editor]
@@ -68,6 +69,7 @@
         today? (= (string/lower-case title)
                   (string/lower-case (date/journal-name)))
         intro? (and (not (state/logged?))
+                    (not (config/local-db? repo))
                     (not config/publishing?)
                     today?)]
     [:div.flex-1.journal.page {:class (if intro? "intro" "")}

src/main/frontend/components/page.cljs

@@ -87,16 +87,15 @@
       (page-blocks-cp repo contents file-path name original-name name true false false nil format))))
 
 (defn presentation
-  [repo page journal?]
-  [:a.opacity-50.hover:opacity-100.ml-4
+  [repo page]
+  [:a.opacity-50.hover:opacity-100
    {:title "Presentation mode (Powered by Reveal.js)"
     :on-click (fn []
                 (state/sidebar-add-block!
                  repo
                  (:db/id page)
                  :page-presentation
-                 {:page page
-                  :journal? journal?}))}
+                 {:page page}))}
    svg/slideshow])
 
 (rum/defc today-queries < rum/reactive
@@ -354,11 +353,16 @@
                          (not block?)
                          (not (state/hide-file?))
                          (not config/publishing?))
-                [:div.text-sm.ml-1.mb-4.flex-1 {:key "page-file"}
-                 [:span.opacity-50 (t :file/file)]
-                 [:a.bg-base-2.p-1.ml-1 {:style {:border-radius 4}
-                                         :href (str "/file/" (util/url-encode file-path))}
-                  file-path]])]
+                [:div.text-sm.ml-1.mb-4.flex-1.inline-flex
+                 {:key "page-file"}
+                 [:span.opacity-50 {:style {:margin-top 2}} (t :file/file)]
+                 [:a.bg-base-2.px-1.ml-1.mr-3 {:style {:border-radius 4}
+                                               :href (str "/file/" (util/url-encode file-path))}
+                  file-path]
+
+                 (when (and (not config/mobile?)
+                            (not journal?))
+                   (presentation repo page))])]
 
              (when (and repo (not block?))
                (let [alias (db/get-page-alias-names repo page-name)]

src/main/frontend/components/reference.cljs

@@ -69,11 +69,11 @@
 
 (rum/defcs unlinked-references-aux
   < rum/reactive db-mixins/query
-    {:will-mount (fn [state]
-                   (let [[page-name n-ref] (:rum/args state)
-                         ref-blocks (db/get-page-unlinked-references page-name)]
-                     (reset! n-ref (count ref-blocks))
-                     (assoc state ::ref-blocks ref-blocks)))}
+  {:will-mount (fn [state]
+                 (let [[page-name n-ref] (:rum/args state)
+                       ref-blocks (db/get-page-unlinked-references page-name)]
+                   (reset! n-ref (count ref-blocks))
+                   (assoc state ::ref-blocks ref-blocks)))}
   [state page-name n-ref]
   (let [ref-blocks (::ref-blocks state)]
     [:div.references-blocks
@@ -100,5 +100,5 @@
             (if @n-ref
               (str @n-ref " Unlinked References")
               "Unlinked References")]
-            (fn [] (unlinked-references-aux page-name n-ref))
+           (fn [] (unlinked-references-aux page-name n-ref))
            true)]]))))

src/main/frontend/components/repo.cljs

@@ -8,11 +8,13 @@
             [frontend.handler.common :as common-handler]
             [frontend.handler.route :as route-handler]
             [frontend.handler.export :as export-handler]
+            [frontend.handler.web.nfs :as nfs-handler]
             [frontend.util :as util]
             [frontend.config :as config]
             [reitit.frontend.easy :as rfe]
             [frontend.version :as version]
             [frontend.components.commit :as commit]
+            [frontend.components.svg :as svg]
             [frontend.context.i18n :as i18n]
             [clojure.string :as string]))
 
@@ -22,159 +24,199 @@
 
 (rum/defc repos < rum/reactive
   []
-  (let [{:keys [repos]} (state/sub :me)
+  (let [repos (->> (state/sub [:me :repos])
+                   (remove #(= (:url %) config/local-repo)))
         repos (util/distinct-by :url repos)]
-    (if (seq repos)
-      [:div#repos
-       [:h1.title "All Repos"]
+    (rum/with-context [[t] i18n/*tongue-context*]
+      (if (seq repos)
+        [:div#repos
+         [:h1.title "All Graphs"]
 
-       [:div.pl-1.content
-        [:div.flex.my-4 {:key "add-button"}
-         (ui/button
-          "Add another repo"
-          :href (rfe/href :repo-add))]
+         [:div.pl-1.content
+          [:div.flex.flex-row.my-4
+           (when (state/logged?)
+             [:div.mr-8
+              (ui/button
+               "Add another git repo"
+               :href (rfe/href :repo-add))])
+           (when (nfs-handler/supported?)
+             [:div.flex.flex-col
+              [:div (ui/button
+                      (t :open-a-directory)
+                      :on-click nfs-handler/ls-dir-files)]
+              [:span.warning.mt-2.text-sm "Warning: this is an experimental feature,"
+               [:br]
+               "please only use it for testing purpose."]])]
+          (for [{:keys [id url] :as repo} repos]
+            (let [local? (config/local-db? url)]
+              [:div.flex.justify-between.mb-1 {:key id}
+               (if local?
+                 [:a
+                  (config/get-local-dir url)]
+                 [:a {:target "_blank"
+                      :href url}
+                  (db/get-repo-path url)])
+               [:div.controls
+                [:a.control {:title (if local?
+                                      "Sync with the local directory"
+                                      "Clone again and re-index the db")
+                             :on-click (fn []
+                                         (if local?
+                                           (nfs-handler/refresh! url)
+                                           (repo-handler/rebuild-index! url))
+                                         (js/setTimeout
+                                          (fn []
+                                            (route-handler/redirect! {:to :home}))
+                                          500))}
+                 "Re-index"]
+                [:a.control.ml-4 {:title "Clone again and re-index the db"
+                                  :on-click (fn []
+                                              (export-handler/export-repo-as-json! (:url repo)))}
+                 "Export as JSON"]
+                [:a.text-gray-400.ml-4 {:on-click (fn []
+                                                    (repo-handler/remove-repo! repo))}
+                 "Unlink"]]]))]
 
-        (for [{:keys [id url] :as repo} repos]
-          [:div.flex.justify-between.mb-1 {:key id}
-           [:a {:target "_blank"
-                :href url}
-            (db/get-repo-path url)]
-           [:div.controls
-            [:a.control {:title "Clone again and re-index the db"
-                         :on-click (fn []
-                                     (repo-handler/rebuild-index! repo)
-                                     (js/setTimeout
-                                      (fn []
-                                        (route-handler/redirect! {:to :home}))
-                                      500))}
-             "Re-index"]
-            [:a.control.ml-4 {:title "Clone again and re-index the db"
-                              :on-click (fn []
-                                          (export-handler/export-repo-as-json! (:url repo)))}
-             "Export as JSON"]
-            [:a.text-gray-400.ml-4 {:on-click (fn []
-                                                (repo-handler/remove-repo! repo))}
-             "Unlink"]]])]
-
-       [:a#download-as-json.hidden]]
-      (widgets/add-repo))))
+         [:a#download-as-json.hidden]]
+        (widgets/add-repo)))))
 
 (rum/defc sync-status < rum/reactive
   {:did-mount (fn [state]
                 (js/setTimeout common-handler/check-changed-files-status 1000)
                 state)}
   []
-  (let [repo (state/get-current-repo)]
-    (when-not (= repo config/local-repo)
-      (let [changed-files (state/sub [:repo/changed-files repo])
-            should-push? (seq changed-files)
-            git-status (state/sub [:git/status repo])
-            pushing? (= :pushing git-status)
-            pulling? (= :pulling git-status)
-            push-failed? (= :push-failed git-status)
-            last-pulled-at (db/sub-key-value repo :git/last-pulled-at)
-            editing? (seq (state/sub :editor/editing?))]
-        [:div.flex-row.flex.items-center
-         (when pushing?
-           [:span.lds-dual-ring.mt-1])
-         (ui/dropdown
-          (fn [{:keys [toggle-fn]}]
-            [:div.cursor.w-2.h-2.sync-status.mr-2
-             {:class (cond
+  (when-let [repo (state/get-current-repo)]
+    (let [nfs-repo? (config/local-db? repo)]
+      (when-not (= repo config/local-repo)
+        (if (and nfs-repo? (nfs-handler/supported?))
+          (let [syncing? (state/sub :graph/syncing?)]
+            [:div.ml-2.mr-1.opacity-70.hover:opacity-100 {:class (if syncing? "loader" "initial")}
+             [:a
+              {:on-click #(nfs-handler/refresh! repo)
+               :title (str "Sync files with the local directory: " (config/get-local-dir repo))}
+              svg/refresh]])
+          (let [changed-files (state/sub [:repo/changed-files repo])
+                should-push? (seq changed-files)
+                git-status (state/sub [:git/status repo])
+                pushing? (= :pushing git-status)
+                pulling? (= :pulling git-status)
+                push-failed? (= :push-failed git-status)
+                last-pulled-at (db/sub-key-value repo :git/last-pulled-at)
+                ;; db-persisted? (state/sub [:db/persisted? repo])
+                editing? (seq (state/sub :editor/editing?))]
+            [:div.flex-row.flex.items-center
+             (when pushing?
+               [:span.lds-dual-ring.mt-1])
+             (ui/dropdown
+              (fn [{:keys [toggle-fn]}]
+                [:div.cursor.w-2.h-2.sync-status.mr-2
+                 {:class (cond
+                           push-failed?
+                           "bg-red-500"
+                           (or
+                            ;; (not db-persisted?)
+                            editing?
+                            should-push? pushing?)
+                           "bg-orange-400"
+                           :else
+                           "bg-green-600")
+                  :style {:border-radius "50%"
+                          :margin-top 2}
+                  :on-mouse-over
+                  (fn [e]
+                    (toggle-fn)
+                    (js/setTimeout common-handler/check-changed-files-status 0))}])
+              (fn [{:keys [toggle-fn]}]
+                (rum/with-context [[t] i18n/*tongue-context*]
+                  [:div.p-2.rounded-md.shadow-xs.bg-base-3.flex.flex-col.sync-content
+                   {:on-mouse-leave toggle-fn}
+                   [:div
+                    [:div
+                     (cond
                        push-failed?
-                       "bg-red-500"
-                       (or editing? should-push? pushing?)
-                       "bg-orange-400"
+                       [:p (t :git/push-failed)]
+                       (and should-push? (seq changed-files))
+                       [:div.changes
+                        [:ul.overflow-y-scroll {:style {:max-height 250}}
+                         (for [file changed-files]
+                           [:li {:key (str "sync-" file)}
+                            [:div.flex.flex-row.justify-between.align-items
+                             [:a {:href (rfe/href :file {:path file})}
+                              file]
+                             [:a.ml-4.text-sm.mt-1
+                              {:on-click (fn [e]
+                                           (export-handler/download-file! file))}
+                              [:span (t :download)]]]])]]
                        :else
-                       "bg-green-600")
-              :style {:border-radius "50%"
-                      :margin-top 2}
-              :on-mouse-over
-              (fn [e]
-                (toggle-fn)
-                (js/setTimeout common-handler/check-changed-files-status 0))}])
-          (fn [{:keys [toggle-fn]}]
-            (rum/with-context [[t] i18n/*tongue-context*]
-              [:div.p-2.rounded-md.shadow-xs.bg-base-3.flex.flex-col.sync-content
-               {:on-mouse-leave toggle-fn}
-               [:div
-                [:div
-                 (cond
-                   push-failed?
-                   [:p (t :git/push-failed)]
-                   (and should-push? (seq changed-files))
-                   [:div.changes
-                    [:ul.overflow-y-scroll {:style {:max-height 250}}
-                     (for [file changed-files]
-                       [:li {:key (str "sync-" file)}
-                        [:div.flex.flex-row.justify-between.align-items
-                         [:a {:href (rfe/href :file {:path file})}
-                          file]
-                         [:a.ml-4.text-sm.mt-1
-                          {:on-click (fn [e]
-                                       (export-handler/download-file! file))}
-                          [:span (t :download)]]]])]]
-                   :else
-                   [:p (t :git/local-changes-synced)])]
-        ;; [:a.text-sm.font-bold {:href "/diff"} "Check diff"]
-                [:div.flex.flex-row.justify-between.align-items.mt-2
-                 (ui/button (t :git/push)
-                            :on-click (fn [] (state/set-modal! commit/add-commit-message)))
-                 (if pushing?
-                   [:span.lds-dual-ring.mt-1])]]
-               [:hr]
-               [:div
-                (when-not (string/blank? last-pulled-at)
-                  [:p {:style {:font-size 12}} (t :git/last-pull)
-                   (str ": " last-pulled-at)])
-                [:div.flex.flex-row.justify-between.align-items
-                 (ui/button (t :git/pull)
-                            :on-click (fn [] (repo-handler/pull-current-repo)))
-                 (if pulling?
-                   [:span.lds-dual-ring.mt-1])]
-                [:a.mt-5.text-sm.opacity-50.block
-                 {:on-click (fn []
-                              (export-handler/export-repo-as-zip! repo))}
-                 (t :repo/download-zip)]
-                [:p.pt-2.text-sm.opacity-50
-                 (t :git/version) (str " " version/version)]]])))]))))
+                       [:p (t :git/local-changes-synced)])]
+                   ;; [:a.text-sm.font-bold {:href "/diff"} "Check diff"]
+                    [:div.flex.flex-row.justify-between.align-items.mt-2
+                     (ui/button (t :git/push)
+                                :on-click (fn [] (state/set-modal! commit/add-commit-message)))
+                     (if pushing?
+                       [:span.lds-dual-ring.mt-1])]]
+                   [:hr]
+                   [:div
+                    (when-not (string/blank? last-pulled-at)
+                      [:p {:style {:font-size 12}} (t :git/last-pull)
+                       (str ": " last-pulled-at)])
+                    [:div.flex.flex-row.justify-between.align-items
+                     (ui/button (t :git/pull)
+                                :on-click (fn [] (repo-handler/pull-current-repo)))
+                     (if pulling?
+                       [:span.lds-dual-ring.mt-1])]
+                    [:a.mt-5.text-sm.opacity-50.block
+                     {:on-click (fn []
+                                  (export-handler/export-repo-as-zip! repo))}
+                     (t :repo/download-zip)]
+                    [:p.pt-2.text-sm.opacity-50
+                     (t :git/version) (str " " version/version)]]])))]))))))
 
 (rum/defc repos-dropdown < rum/reactive
   [head? on-click]
-  (let [current-repo (state/sub :git/current-repo)
-        logged? (state/logged?)
-        local-repo? (= current-repo config/local-repo)
-        get-repo-name (fn [repo]
-                        (if head?
-                          (db/get-repo-path repo)
-                          (util/take-at-most (db/get-repo-name repo) 20)))]
-    (when logged?
-      (if current-repo
-        (let [repos (state/sub [:me :repos])]
-          (if (> (count repos) 1)
-            (ui/dropdown-with-links
-             (fn [{:keys [toggle-fn]}]
-               [:a#repo-switch {:on-click toggle-fn}
-                [:span (get-repo-name current-repo)]
-                [:span.dropdown-caret.ml-1 {:style {:border-top-color "#6b7280"}}]])
-             (mapv
-              (fn [{:keys [id url]}]
-                {:title (get-repo-name url)
-                 :options {:on-click (fn []
-                                       (repo-handler/push-if-auto-enabled! (state/get-current-repo))
-                                       (state/set-current-repo! url)
-                                       (when-not (= :draw (state/get-current-route))
-                                         (route-handler/redirect-to-home!))
-                                       (when on-click
-                                         (on-click url)))}})
-              (remove (fn [repo]
-                        (= current-repo (:url repo)))
-                      repos))
-             {:modal-class (util/hiccup->class
-                            "origin-top-right.absolute.left-0.mt-2.w-48.rounded-md.shadow-lg ")})
-            (if local-repo?
+  (when-let [current-repo (state/sub :git/current-repo)]
+    (let [logged? (state/logged?)
+          local-repo? (= current-repo config/local-repo)
+          get-repo-name (fn [repo]
+                          (if (config/local-db? repo)
+                            (config/get-local-dir repo)
+                            (if head?
+                              (db/get-repo-path repo)
+                              (util/take-at-most (db/get-repo-name repo) 20))))]
+      (let [repos (->> (state/sub [:me :repos])
+                       (remove (fn [r] (= config/local-repo (:url r)))))]
+        (cond
+          (> (count repos) 1)
+          (ui/dropdown-with-links
+           (fn [{:keys [toggle-fn]}]
+             [:a#repo-switch {:on-click toggle-fn}
               [:span (get-repo-name current-repo)]
+              [:span.dropdown-caret.ml-1 {:style {:border-top-color "#6b7280"}}]])
+           (mapv
+            (fn [{:keys [id url]}]
+              {:title (get-repo-name url)
+               :options {:on-click (fn []
+                                     (repo-handler/push-if-auto-enabled! (state/get-current-repo))
+
+                                     (state/set-current-repo! url)
+                                     (when-not (= :draw (state/get-current-route))
+                                       (route-handler/redirect-to-home!))
+                                     (when on-click
+                                       (on-click url)))}})
+            (remove (fn [repo]
+                      (= current-repo (:url repo)))
+                    repos))
+           {:modal-class (util/hiccup->class
+                          "origin-top-right.absolute.left-0.mt-2.w-48.rounded-md.shadow-lg ")})
+
+          (and current-repo (not local-repo?))
+          (let [repo-name (get-repo-name current-repo)]
+            (if (config/local-db? current-repo)
+              repo-name
               [:a
                {:href current-repo
                 :target "_blank"}
-               (get-repo-name current-repo)])))))))
+               repo-name]))
+
+          :else
+          nil)))))

src/main/frontend/components/sidebar.cljs

@@ -118,8 +118,6 @@
       [:div.cp__sidebar-main-content
        {:data-is-global-graph-pages global-graph-pages?
         :data-is-full-width (or global-graph-pages?
-                                (and (not logged?)
-                                     home?)
                                 (contains? #{:all-files :all-pages} route-name))}
        (cond
          (not indexeddb-support?)
@@ -174,22 +172,22 @@
         current-repo (state/sub :git/current-repo)
         latest-journals (db/get-latest-journals (state/get-current-repo) journals-length)
         preferred-format (state/sub [:me :preferred_format])
-        logged? (:name me)
-        token (state/sub :encrypt/token)
-        ;; TODO: remove this
-        daily-migrating? (state/sub [:daily/migrating?])]
+        logged? (:name me)]
     (rum/with-context [[t] i18n/*tongue-context*]
       [:div.max-w-7xl.mx-auto
        (cond
-         daily-migrating?
-         (ui/loading "Migrating to daily notes")
-
          (and default-home
               (= :home (state/get-current-route))
               (not (state/route-has-p?)))
          (route-handler/redirect! {:to :page
                                    :path-params {:name (:page default-home)}})
 
+         importing-to-db?
+         (ui/loading (t :parsing-files))
+
+         loading-files?
+         (ui/loading (t :loading-files))
+
          (and (not logged?) (seq latest-journals))
          (journal/journals latest-journals)
 
@@ -206,12 +204,6 @@
          (seq latest-journals)
          (journal/journals latest-journals)
 
-         importing-to-db?
-         (ui/loading (t :parsing-files))
-
-         loading-files?
-         (ui/loading (t :loading-files))
-
          (and logged? (empty? (:repos me)))
          (widgets/add-repo)
 

src/main/frontend/components/sidebar.css

@@ -18,7 +18,7 @@
 .cp__sidebar-main-content {
     padding: 3rem 1.5rem;
     margin: 0 auto;
-    max-width: 640px;
+    max-width: 700px;
 }
 
 @media (max-width: 640px) {

src/main/frontend/components/svg.cljs

@@ -85,6 +85,10 @@
       :stroke "currentColor"
       :d d}]]))
 
+(def refresh
+  (hero-icon "M4 4V9H4.58152M19.9381 11C19.446 7.05369 16.0796 4 12 4C8.64262 4 5.76829 6.06817 4.58152 9M4.58152 9H9M20 20V15H19.4185M19.4185 15C18.2317 17.9318 15.3574 20 12 20C7.92038 20 4.55399 16.9463 4.06189 13M19.4185 15H15"
+             {:fill "none"}))
+
 (def user
   [:svg
    {:stroke-linejoin "round"
@@ -122,6 +126,17 @@
      :x1 "10.5"}]])
 
 (def graph-sm [:div {:style {:transform "rotate(90deg)"}} (hero-icon "M8.684 13.342C8.886 12.938 9 12.482 9 12c0-.482-.114-.938-.316-1.342m0 2.684a3 3 0 110-2.684m0 2.684l6.632 3.316m-6.632-6l6.632-3.316m0 0a3 3 0 105.367-2.684 3 3 0 00-5.367 2.684zm0 9.316a3 3 0 105.368 2.684 3 3 0 00-5.368-2.684z" {:height "16" :width "16"})])
+
+(def folder-add
+  [:svg
+   {:stroke "currentColor", :view-box "0 0 24 24", :fill "none" :width 24 :height 24 :display "inline-block"}
+   [:path
+    {:d
+     "M9 13h6m-3-3v6m-9 1V7a2 2 0 012-2h6l2 2h6a2 2 0 012 2v8a2 2 0 01-2 2H5a2 2 0 01-2-2z",
+     :stroke-width "2",
+     :stroke-linejoin "round",
+     :stroke-linecap "round"}]])
+
 (def folder (hero-icon "M3 7v10a2 2 0 002 2h14a2 2 0 002-2V9a2 2 0 00-2-2h-6l-2-2H5a2 2 0 00-2 2z"))
 (def folder-sm (hero-icon "M3 7v10a2 2 0 002 2h14a2 2 0 002-2V9a2 2 0 00-2-2h-6l-2-2H5a2 2 0 00-2 2z" {:height "16" :width "16"}))
 (def pages-sm [:svg {:viewbox "0 0 20 20", :fill "currentColor", :height "16", :width "16"}
@@ -327,10 +342,9 @@
 (def slideshow
   [:svg
    {:view-box "0 0 24 24"
-    :height 23
-    :width 23
-    :fill "currentColor"
-    :display "inline-block"}
+    :height 24
+    :width 24
+    :fill "currentColor"}
    [:path
     {:d "M10 8v8l5-4-5-4zm9-5H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zm0 16H5V5h14v14z"}]])
 
@@ -395,3 +409,6 @@
      :stroke-width "2"
      :stroke-linejoin "round"
      :stroke-linecap "round"}]])
+
+(def online
+  (hero-icon "M8.111 16.404a5.5 5.5 0 017.778 0M12 20h.01m-7.08-7.071c3.904-3.905 10.236-3.905 14.141 0M1.394 9.393c5.857-5.857 15.355-5.857 21.213 0"))

src/main/frontend/config.cljs

@@ -226,6 +226,15 @@
     "md"
     (name format)))
 
+(defn get-file-format
+  [extension]
+  (case (keyword extension)
+    :markdown
+    :markdown
+    :md
+    :markdown
+    (keyword extension)))
+
 (defn default-empty-block
   ([format]
    (default-empty-block format 2))
@@ -269,3 +278,15 @@
 (def markers
   #{"now" "later" "todo" "doing" "done" "wait" "waiting"
     "canceled" "cancelled" "started" "in-progress"})
+
+(defonce idb-db-prefix "logseq-db/")
+(defonce local-db-prefix "logseq_local_")
+(defonce local-handle-prefix (str "handle/" local-db-prefix))
+
+(defn local-db?
+  [s]
+  (string/starts-with? s local-db-prefix))
+
+(defn get-local-dir
+  [s]
+  (string/replace s local-db-prefix ""))

src/main/frontend/db.cljs

@@ -11,7 +11,6 @@
             [clojure.set :as set]
             [frontend.utf8 :as utf8]
             [frontend.config :as config]
-            ["localforage" :as localforage]
             [promesa.core :as p]
             [cljs.reader :as reader]
             [cljs-time.core :as t]
@@ -21,36 +20,13 @@
             [frontend.extensions.sci :as sci]
             [frontend.db-schema :as db-schema]
             [clojure.core.async :as async]
-            [frontend.storage :as storage]
             [lambdaisland.glogi :as log]
-            [goog.object :as gobj]))
-
-;; offline db
-(def store-name "dbs")
-(.config localforage
-         #js
-          {:name "logseq-datascript"
-           :version 1.0
-           :storeName store-name})
-
-(defonce localforage-instance (.createInstance localforage store-name))
+            [frontend.idb :as idb]))
 
 ;; Query atom of map of Key ([repo q inputs]) -> atom
 ;; TODO: replace with LRUCache, only keep the latest 20 or 50 items?
 (defonce query-state (atom {}))
 
-(defn clear-idb!
-  []
-  (p/let [_ (.clear localforage-instance)
-          dbs (js/window.indexedDB.databases)]
-    (doseq [db dbs]
-      (js/window.indexedDB.deleteDatabase (gobj/get db "name")))))
-
-(defn clear-local-storage-and-idb!
-  []
-  (storage/clear)
-  (clear-idb!))
-
 (defn get-repo-path
   [url]
   (if (util/starts-with? url "http")
@@ -61,7 +37,7 @@
 (defn datascript-db
   [repo]
   (when repo
-    (str "logseq-db/" (get-repo-path repo))))
+    (str config/idb-db-prefix (get-repo-path repo))))
 
 (defn datascript-files-db
   [repo]
@@ -70,11 +46,11 @@
 
 (defn remove-db!
   [repo]
-  (.removeItem localforage-instance (datascript-db repo)))
+  (idb/remove-item! (datascript-db repo)))
 
 (defn remove-files-db!
   [repo]
-  (.removeItem localforage-instance (datascript-files-db repo)))
+  (idb/remove-item! (datascript-files-db repo)))
 
 (def react util/react)
 
@@ -110,6 +86,13 @@
   (swap! conns dissoc (datascript-db repo))
   (swap! conns dissoc (datascript-files-db repo)))
 
+(defn get-tx-id [tx-report]
+  (get-in tx-report [:tempids :db/current-tx]))
+
+(defn get-max-tx-id
+  [db]
+  (:max-tx db))
+
 ;; transit serialization
 
 (defn db->string [db]
@@ -124,13 +107,16 @@
 (defn string->db [s]
   (dt/read-transit-str s))
 
-;; persisting DB between page reloads
-(defn persist [repo db files-db?]
-  (.setItem localforage-instance
-            (if files-db?
-              (datascript-files-db repo)
-              (datascript-db repo))
-            (db->string db)))
+;; persisting DBs between page reloads
+(defn persist! [repo]
+  (let [file-key (datascript-files-db repo)
+        non-file-key (datascript-db repo)
+        file-db (d/db (get-files-conn repo))
+        non-file-db (d/db (get-conn repo false))]
+    (p/let [_ (idb/set-item! file-key (db->string file-db))
+            _ (idb/set-item! non-file-key (db->string non-file-db))]
+      (state/set-last-persist-transact-id! repo true (get-max-tx-id file-db))
+      (state/set-last-persist-transact-id! repo false (get-max-tx-id non-file-db)))))
 
 (defn reset-conn! [conn db]
   (reset! conn db))
@@ -556,9 +542,6 @@
           (group-by-page result)))
       result)))
 
-(defn get-tx-id [tx-report]
-  (get-in tx-report [:tempids :db/current-tx]))
-
 (defn transact!
   ([tx-data]
    (transact! (state/get-current-repo) tx-data))
@@ -568,8 +551,19 @@
                         (remove nil?))]
        (when (seq tx-data)
          (when-let [conn (get-conn repo-url false)]
-           (let [tx-report (d/transact! conn (vec tx-data))]
-             (state/mark-repo-as-changed! repo-url (get-tx-id tx-report)))))))))
+           (d/transact! conn (vec tx-data))))))))
+
+(defn transact-files-db!
+  ([tx-data]
+   (transact! (state/get-current-repo) tx-data))
+  ([repo-url tx-data]
+   (when-not config/publishing?
+     (let [tx-data (->> (util/remove-nils tx-data)
+                        (remove nil?)
+                        (map #(dissoc % :file/handle :file/type)))]
+       (when (seq tx-data)
+         (when-let [conn (get-files-conn repo-url)]
+           (d/transact! conn (vec tx-data))))))))
 
 (defn get-key-value
   ([key]
@@ -594,42 +588,41 @@
   (when-not config/publishing?
     (try
       (let [repo-url (or repo-url (state/get-current-repo))
-           tx-data (->> (util/remove-nils tx-data)
-                        (remove nil?))
-           get-conn (fn [] (if files-db?
-                             (get-files-conn repo-url)
-                             (get-conn repo-url false)))]
-       (when (and (seq tx-data) (get-conn))
-         (let [tx-result (profile "Transact!" (d/transact! (get-conn) (vec tx-data)))
-               _ (state/mark-repo-as-changed! repo-url (get-tx-id tx-result))
-               db (:db-after tx-result)
-               handler-keys (get-handler-keys handler-opts)]
-           (doseq [handler-key handler-keys]
-             (let [handler-key (vec (cons repo-url handler-key))]
-               (when-let [cache (get @query-state handler-key)]
-                 (let [{:keys [query inputs transform-fn query-fn inputs-fn]} cache]
-                   (when (or query query-fn)
-                     (let [new-result (->
-                                       (cond
-                                         query-fn
-                                         (profile
-                                          "Query:"
-                                          (doall (query-fn db)))
-
-                                         inputs-fn
-                                         (let [inputs (inputs-fn)]
-                                           (apply d/q query db inputs))
-
-                                         (keyword? query)
-                                         (get-key-value repo-url query)
-
-                                         (seq inputs)
-                                         (apply d/q query db inputs)
-
-                                         :else
-                                         (d/q query db))
-                                       transform-fn)]
-                       (set-new-result! handler-key new-result))))))))))
+            tx-data (->> (util/remove-nils tx-data)
+                         (remove nil?))
+            get-conn (fn [] (if files-db?
+                              (get-files-conn repo-url)
+                              (get-conn repo-url false)))]
+        (when (and (seq tx-data) (get-conn))
+          (let [tx-result (profile "Transact!" (d/transact! (get-conn) (vec tx-data)))
+                db (:db-after tx-result)
+                handler-keys (get-handler-keys handler-opts)]
+            (doseq [handler-key handler-keys]
+              (let [handler-key (vec (cons repo-url handler-key))]
+                (when-let [cache (get @query-state handler-key)]
+                  (let [{:keys [query inputs transform-fn query-fn inputs-fn]} cache]
+                    (when (or query query-fn)
+                      (let [new-result (->
+                                        (cond
+                                          query-fn
+                                          (profile
+                                           "Query:"
+                                           (doall (query-fn db)))
+
+                                          inputs-fn
+                                          (let [inputs (inputs-fn)]
+                                            (apply d/q query db inputs))
+
+                                          (keyword? query)
+                                          (get-key-value repo-url query)
+
+                                          (seq inputs)
+                                          (apply d/q query db inputs)
+
+                                          :else
+                                          (d/q query db))
+                                        transform-fn)]
+                        (set-new-result! handler-key new-result))))))))))
       (catch js/Error e
         ;; FIXME: check error type and notice user
         (log/error :db/transact! e)))))
@@ -904,7 +897,8 @@
     (transact-react!
      repo
      [{:file/path path
-       :file/content content}]
+       :file/content content
+       :file/last-modified-at (util/time-ms)}]
      {:key [:file/content path]
       :files-db? true})))
 
@@ -931,13 +925,24 @@
   (when-let [conn (get-files-conn repo)]
     (->>
      (d/q
-       '[:find ?path ?content
-         :where
-         [?file :file/path ?path]
-         [?file :file/content ?content]]
-       @conn)
+      '[:find ?path ?content
+        :where
+        [?file :file/path ?path]
+        [?file :file/content ?content]]
+      @conn)
      (into {}))))
 
+(defn get-files-full
+  [repo]
+  (when-let [conn (get-files-conn repo)]
+    (->>
+     (d/q
+      '[:find (pull ?file [*])
+        :where
+        [?file :file/path]]
+      @conn)
+     (flatten))))
+
 (defn get-custom-css
   []
   (get-file "logseq/custom.css"))
@@ -960,12 +965,9 @@
         ffirst)))))
 
 (defn reset-contents-and-blocks!
-  [repo-url contents blocks-pages delete-files delete-blocks]
-  (let [files (doall
-               (map (fn [[file content]]
-                      (set-file-content! repo-url file content)
-                      {:file/path file})
-                    contents))
+  [repo-url files blocks-pages delete-files delete-blocks]
+  (transact-files-db! repo-url files)
+  (let [files (map #(select-keys % [:file/path]) files)
         all-data (-> (concat delete-files delete-blocks files blocks-pages)
                      (util/remove-nils))]
     (transact! repo-url all-data)))
@@ -1504,22 +1506,24 @@
          [[(get-page-name file ast) blocks]])))))
 
 (defn extract-all-blocks-pages
-  [repo-url contents]
-  (let [result (->> contents
-                    (map
-                      (fn [[file content] contents]
-                        (println "Parsing : " file)
-                        (when content
-                          (let [utf8-content (utf8/encode content)]
-                            (extract-blocks-pages repo-url file content utf8-content)))))
-                    (remove empty?))
-        [pages block-ids blocks] (apply map concat result)
-        block-ids-set (set block-ids)
-        blocks (map (fn [b]
-                      (-> b
-                          (update :block/ref-blocks #(set/intersection (set %) block-ids-set))
-                          (update :block/embed-blocks #(set/intersection (set %) block-ids-set)))) blocks)]
-    (apply concat [pages block-ids blocks])))
+  [repo-url files]
+  (when (seq files)
+    (let [result (->> files
+                      (map
+                       (fn [{:file/keys [path content]} contents]
+                         (println "Parsing : " path)
+                         (when content
+                           (let [utf8-content (utf8/encode content)]
+                             (extract-blocks-pages repo-url path content utf8-content)))))
+                      (remove empty?))]
+      (when (seq result)
+        (let [[pages block-ids blocks] (apply map concat result)
+              block-ids-set (set block-ids)
+              blocks (map (fn [b]
+                            (-> b
+                                (update :block/ref-blocks #(set/intersection (set %) block-ids-set))
+                                (update :block/embed-blocks #(set/intersection (set %) block-ids-set)))) blocks)]
+          (apply concat [pages block-ids blocks]))))))
 
 ;; TODO: compare blocks
 (defn reset-file!
@@ -1874,17 +1878,74 @@
       (state/set-config! repo-url config)
       config)))
 
+(defonce persistent-jobs (atom {}))
+
+(defn clear-repo-persistent-job!
+  [repo]
+  (when-let [old-job (get @persistent-jobs repo)]
+    (js/clearTimeout old-job)))
+
+(defn- persist-if-idle!
+  [repo]
+  (clear-repo-persistent-job! repo)
+  (let [job (js/setTimeout
+             (fn []
+               (if (and (state/input-idle? repo)
+                        (state/db-idle? repo))
+                 (do
+                   (persist! repo)
+                   ;; (state/set-db-persisted! repo true)
+)
+                 (let [job (get persistent-jobs repo)]
+                   (persist-if-idle! repo))))
+             3000)]
+    (swap! persistent-jobs assoc repo job)))
+
+;; only save when user's idle
+(defn- repo-listen-to-tx!
+  [repo conn files-db?]
+  (d/listen! conn :persistence
+             (fn [tx-report]
+               (let [tx-id (get-tx-id tx-report)]
+                 (state/set-last-transact-time! repo (util/time-ms))
+                 ;; (state/persist-transaction! repo files-db? tx-id (:tx-data tx-report))
+                 (persist-if-idle! repo)))))
+
+(defn- listen-and-persist!
+  [repo]
+  (when-let [conn (get-files-conn repo)]
+    (repo-listen-to-tx! repo conn true))
+  (when-let [conn (get-conn repo false)]
+    (repo-listen-to-tx! repo conn false)))
+
 (defn start-db-conn!
-  [me repo]
-  (let [files-db-name (datascript-files-db repo)
-        files-db-conn (d/create-conn db-schema/files-db-schema)
-        db-name (datascript-db repo)
-        db-conn (d/create-conn db-schema/schema)]
-    (swap! conns assoc files-db-name files-db-conn)
-    (swap! conns assoc db-name db-conn)
-    (d/transact! db-conn [{:schema/version db-schema/version}])
-    (when me
-      (d/transact! db-conn [(me-tx (d/db db-conn) me)]))))
+  ([me repo]
+   (start-db-conn! me repo {}))
+  ([me repo {:keys [db-type]}]
+   (let [files-db-name (datascript-files-db repo)
+         files-db-conn (d/create-conn db-schema/files-db-schema)
+         db-name (datascript-db repo)
+         db-conn (d/create-conn db-schema/schema)]
+     (swap! conns assoc files-db-name files-db-conn)
+     (swap! conns assoc db-name db-conn)
+     (d/transact! db-conn [(cond-> {:schema/version db-schema/version}
+                             db-type
+                             (assoc :db/type db-type))])
+     (when me
+       (d/transact! db-conn [(me-tx (d/db db-conn) me)]))
+
+     (listen-and-persist! repo))))
+
+(defonce tx-data-debug (atom nil))
+(defn with-latest-txs!
+  [db repo file?]
+  (let [txs (state/get-repo-latest-txs repo file?)
+        tx-data (when (seq txs) (map :tx-data txs))]
+    (if (seq tx-data)
+      (do
+        (swap! tx-data-debug assoc file? tx-data)
+        (d/db-with db tx-data))
+      db)))
 
 (defn restore!
   [{:keys [repos] :as me} restore-config-handler]
@@ -1892,10 +1953,11 @@
     (doall
      (for [{:keys [url]} repos]
        (let [repo url
+
              db-name (datascript-files-db repo)
              db-conn (d/create-conn db-schema/files-db-schema)]
          (swap! conns assoc db-name db-conn)
-         (p/let [stored (-> (.getItem localforage-instance db-name)
+         (p/let [stored (-> (idb/get-item db-name)
                             (p/then (fn [result]
                                       result))
                             (p/catch (fn [error]
@@ -1908,14 +1970,15 @@
                  db-conn (d/create-conn db-schema/schema)
                  _ (d/transact! db-conn [{:schema/version db-schema/version}])
                  _ (swap! conns assoc db-name db-conn)
-                 stored (.getItem localforage-instance db-name)
+                 stored (idb/get-item db-name)
                  _ (if stored
                      (let [stored-db (string->db stored)
                            attached-db (d/db-with stored-db [(me-tx stored-db me)])]
                        (reset-conn! db-conn attached-db))
                      (when logged?
                        (d/transact! db-conn [(me-tx (d/db db-conn) me)])))]
-           (restore-config-handler repo)))))))
+           (restore-config-handler repo)
+           (listen-and-persist! repo)))))))
 
 (defn- build-edges
   [edges]
@@ -2447,6 +2510,14 @@
             datoms (d/datoms filtered-db :eavt)]
         @(d/conn-from-datoms datoms db-schema/schema)))))
 
+(defn get-db-type
+  [repo]
+  (get-key-value repo :db/type))
+
+(defn local-native-fs?
+  [repo]
+  (= :local-native-fs (get-db-type repo)))
+
 ;; shortcut for query a block with string ref
 (defn qb
   [string-id]

src/main/frontend/db_schema.cljs

@@ -4,13 +4,17 @@
 
 (def files-db-schema
   {:file/path {:db/unique :db.unique/identity}
-   :file/content {}})
+   :file/content {}
+   :file/last-modified-at {}
+   :file/size {}
+   :file/handle {}})
 
 ;; A page can corresponds to multiple files (same title),
 ;; a month journal file can have multiple pages,
 ;; also, each block can be treated as a page too.
 (def schema
-  {:schema/version {}
+  {:schema/version  {}
+   :db/type         {}
    :db/ident        {:db/unique :db.unique/identity}
 
    ;; user

src/main/frontend/dicts.cljs

@@ -67,13 +67,13 @@ title: How to take dummy notes?
 
 ## Hello, I'm a block!
 :PROPERTIES:
-:custom_id: 5f713e91-8a3c-4b04-a33a-c39482428e2d
+:id: 5f713e91-8a3c-4b04-a33a-c39482428e2d
 :END:
 ### I'm a child block!
 ### I'm another child block!
 ## Hey, I'm another block!
 :PROPERTIES:
-:custom_id: 5f713ea8-8cba-403d-ac00-9964b1ec7190
+:id: 5f713ea8-8cba-403d-ac00-9964b1ec7190
 :END:
 "
         :on-boarding/title "Hi, welcome to Logseq!"
@@ -301,7 +301,9 @@ title: How to take dummy notes?
         :new-file "New file"
         :graph "Graph"
         :publishing "Publishing"
+        :export "Export public pages"
         :all-repos "All repos"
+        :all-graphs "All graphs"
         :all-pages "All pages"
         :all-files "All files"
         :all-journals "All journals"
@@ -316,7 +318,6 @@ title: How to take dummy notes?
         :parsing-files "Parsing files"
         :loading-files "Loading files"
         :login-github "Login with Github"
-        :excalidraw-title "Draw with Excalidraw"
         :go-to "Go to "
         :or "or"
         :download "Download"
@@ -324,7 +325,9 @@ title: How to take dummy notes?
         :language "Language"
         :white "Light"
         :dark "Dark"
-        :remove-background "Remove background"}
+        :remove-background "Remove background"
+        :open "Open"
+        :open-a-directory "Open a local directory"}
 
    :fr {:help/about "A propos de Logseq"
         :help/bug "Signaler une anomalie"
@@ -517,7 +520,6 @@ title: How to take dummy notes?
         :parsing-files "Analyse des fichiers"
         :loading-files "Chargement des fichiers"
         :login-github "S'authentifier avec Github"
-        :excalidraw-title "Dessiner avec Excalidraw"
         :go-to "Aller à "
         :or "ou"
         :download "Télécharger"
@@ -749,7 +751,9 @@ title: How to take dummy notes?
            :new-page "新页面"
            :new-file "新文件"
            :graph "图谱"
-           :publishing "发布/下载 HTML 文件"
+           :publishing "发布"
+           :export "导出公开页面"
+           :all-graphs "所有库"
            :all-repos "所有库"
            :all-pages "所有页面"
            :all-files "所有文件"
@@ -764,7 +768,6 @@ title: How to take dummy notes?
            :parsing-files "正在解析文件"
            :loading-files "正在加载文件"
            :login-github "用 Github 登录"
-           :excalidraw-title "用 Excalidraw 画图"
            :go-to "转到"
            :or "或"
            :download "下载"
@@ -772,7 +775,9 @@ title: How to take dummy notes?
            :language "语言"
            :white "亮色"
            :dark "暗黑"
-           :remove-background "去除背景"}
+           :remove-background "去除背景"
+           :open "打开"
+           :open-a-directory "打开本地文件夹"}
 
    :zh-Hant {:on-boarding/title "你好，歡迎使用 Logseq！"
              :on-boarding/sharing "分享"
@@ -1008,7 +1013,6 @@ title: How to take dummy notes?
              :parsing-files "正在解析文件"
              :loading-files "正在加載文件"
              :login-github "用 Github 登錄"
-             :excalidraw-title "用 Excalidraw 畫圖"
              :go-to "轉到"
              :or "或"
              :download "下載"
@@ -1246,7 +1250,6 @@ title: How to take dummy notes?
         :parsing-files "Lêer ontleding"
         :loading-files "Laai lêers"
         :login-github "Aantekening deur Github"
-        :excalidraw-title "Teken met Excalidraw"
         :go-to "Gaan na "
         :or "of"
         :download "Laai af"

src/main/frontend/diff.cljs

@@ -9,6 +9,19 @@
   (-> ((gobj/get jsdiff "diffLines") s1 s2)
       bean/->clj))
 
+(defn diff-words
+  [s1 s2]
+  (-> ((gobj/get jsdiff "diffWords") s1 s2)
+      bean/->clj))
+
+(defn removed?
+  [s1 s2]
+  (when (and s1 s2)
+    (let [diff-result (diff-words s1 s2)]
+      (->> diff-result
+           (some :removed)
+           (boolean)))))
+
 ;; (find-position "** hello _w_" "hello w")
 (defn find-position
   [markup text]

src/main/frontend/fs.cljs

@@ -1,53 +1,259 @@
 (ns frontend.fs
-  (:require [frontend.util :as util]))
+  (:require [frontend.util :as util :refer-macros [profile]]
+            [frontend.config :as config]
+            [clojure.string :as string]
+            [frontend.idb :as idb]
+            [promesa.core :as p]
+            [goog.object :as gobj]
+            [frontend.diff :as diff]
+            [clojure.set :as set]
+            [lambdaisland.glogi :as log]
+            ["/frontend/utils" :as utils]))
 
-(defn mkdir
+;; We need to cache the file handles in the memory so that
+;; the browser will not keep asking permissions.
+(defonce nfs-file-handles-cache (atom {}))
+
+(defn get-nfs-file-handle
+  [handle-path]
+  (get @nfs-file-handles-cache handle-path))
+
+(defn add-nfs-file-handle!
+  [handle-path handle]
+  (swap! nfs-file-handles-cache assoc handle-path handle))
+
+(defn remove-nfs-file-handle!
+  [handle-path]
+  (swap! nfs-file-handles-cache dissoc handle-path))
+
+;; TODO:
+;; We need to support several platforms:
+;; 1. Chrome native file system API (lighting-fs wip)
+;; 2. IndexedDB (lighting-fs)
+;; 3. NodeJS
+#_(defprotocol Fs
+    (mkdir! [this dir])
+    (readdir! [this dir])
+    (unlink! [this path opts])
+    (rename! [this old-path new-path])
+    (rmdir! [this dir])
+    (read-file [dir path option])
+    (write-file! [dir path content])
+    (stat [dir path]))
+
+(defn local-db?
   [dir]
-  (when (and dir js/window.pfs)
-    (js/window.pfs.mkdir dir)))
+  (and (string? dir)
+       (config/local-db? (subs dir 1))))
 
-(defn mkdir-if-not-exists
+(defn mkdir
   [dir]
-  (when (and dir js/window.pfs)
-    (util/p-handle
-     (js/window.pfs.stat dir)
-     (fn [_stat])
-     (fn [_error] (js/window.pfs.mkdir dir)))))
+  (cond
+    (local-db? dir)
+    (let [[root new-dir] (rest (string/split dir "/"))
+          root-handle (str "handle/" root)]
+      (p/let [handle (idb/get-item root-handle)]
+        (when handle (utils/verifyPermission handle true))
+        (when (and handle new-dir
+                   (not (string/blank? new-dir)))
+          (-> (p/let [handle (.getDirectoryHandle ^js handle new-dir
+                                                  #js {:create true})
+                      handle-path (str root-handle "/" new-dir)
+                      _ (idb/set-item! handle-path handle)]
+                (add-nfs-file-handle! handle-path handle)
+                (println "Stored handle: " (str root-handle "/" new-dir)))
+              (p/catch (fn [error]
+                         (println "mkdir error: " error ", dir: " dir)
+                         (js/console.error error)))))))
+
+    (and dir js/window.pfs)
+    (js/window.pfs.mkdir dir)
+
+    :else
+    (println (str "mkdir " dir " failed"))))
 
 (defn readdir
   [dir]
-  (when (and dir js/window.pfs)
-    (js/window.pfs.readdir dir)))
+  (cond
+    (local-db? dir)
+    (let [prefix (str "handle/" dir)
+          cached-files (keys @nfs-file-handles-cache)]
+      (p/resolved
+       (->> (filter #(string/starts-with? % (str prefix "/")) cached-files)
+            (map (fn [path]
+                   (string/replace path prefix ""))))))
+
+    (and dir js/window.pfs)
+    (js/window.pfs.readdir dir)
+
+    :else
+    nil))
 
 (defn unlink
   [path opts]
-  (js/window.pfs.unlink path opts))
+  (cond
+    (local-db? path)
+    (let [[dir basename] (util/get-dir-and-basename path)
+          handle-path (str "handle" path)]
+      (->
+       (p/let [handle (idb/get-item (str "handle" dir))
+               _ (idb/remove-item! handle-path)]
+         (when handle
+           (.removeEntry ^js handle basename))
+         (remove-nfs-file-handle! handle-path))
+       (p/catch (fn [error]
+                  (log/error :unlink/path {:path path
+                                           :error error})))))
 
-(defn rename
-  [old-path new-path]
-  (js/window.pfs.rename old-path new-path))
+    :else
+    (js/window.pfs.unlink path opts)))
 
 (defn rmdir
+  "Remove the directory recursively."
   [dir]
-  (js/window.workerThread.rimraf dir))
+  (cond
+    (local-db? dir)
+    nil
+
+    :else
+    (js/window.workerThread.rimraf dir)))
 
 (defn read-file
-  [dir path]
-  (js/window.pfs.readFile (str dir "/" path)
-                          (clj->js {:encoding "utf8"})))
+  ([dir path]
+   (read-file dir path (clj->js {:encoding "utf8"})))
+  ([dir path option]
+   (cond
+     (local-db? dir)
+     (let [handle-path (str "handle" dir "/" path)]
+       (p/let [handle (idb/get-item handle-path)
+               local-file (and handle (.getFile handle))]
+         (and local-file (.text local-file))))
 
-(defn read-file-2
-  [dir path]
-  (js/window.pfs.readFile (str dir "/" path)
-                          (clj->js {})))
+     :else
+     (js/window.pfs.readFile (str dir "/" path) option))))
+
+(defn diff-removed?
+  [format s1 s2]
+  (when (and s1 s2)
+    (let [diff-result (diff/diff-words s1 s2)
+          block-pattern (config/get-block-pattern format)]
+      (some (fn [{:keys [removed value]}]
+              (and removed
+                   value
+                   ;; FIXME: not sure why this happened, it might be related to
+                   ;; the async block operations (inserting blocks)
+                   (not (set/superset? #{"#" "\n"} (set (distinct value))))))
+            diff-result))))
 
 (defn write-file
-  [dir path content]
-  (and js/window.pfs (js/window.pfs.writeFile (str dir "/" path) content)))
+  ([dir path content]
+   (write-file dir path content nil))
+  ([dir path content old-content]
+   (cond
+     (local-db? dir)
+     (let [parts (string/split path "/")
+           basename (last parts)
+           sub-dir (->> (butlast parts)
+                        (remove string/blank?)
+                        (string/join "/"))
+           sub-dir-handle-path (str "handle/"
+                                    (subs dir 1)
+                                    (if sub-dir
+                                      (str "/" sub-dir)))
+           handle-path (if (= "/" (last sub-dir-handle-path))
+                         (subs sub-dir-handle-path 0 (dec (count sub-dir-handle-path)))
+                         sub-dir-handle-path)
+           basename-handle-path (str handle-path "/" basename)]
+       (p/let [file-handle (idb/get-item basename-handle-path)]
+         (add-nfs-file-handle! basename-handle-path file-handle)
+         (if file-handle
+           (p/let [local-file (.getFile file-handle)
+                   local-content (.text local-file)]
+             (let [format (-> (util/get-file-ext path)
+                              (config/get-file-format))]
+               (if (and local-content old-content
+                        ;; To prevent data loss, it's not enough to just compare using `=`.
+                        ;; Also, we need to benchmark the performance of `diff/diff-words `
+                        (not (diff-removed?
+                              format
+                              (string/trim local-content)
+                              (string/trim old-content))))
+                 (do
+                   (utils/verifyPermission file-handle true)
+                   (utils/writeFile file-handle content))
+                 (js/alert (str "The file has been modified in your local disk! File path: " path
+                                ", save your changes and click the refresh button to reload it.")))))
+           ;; create file handle
+           (->
+            (p/let [handle (idb/get-item handle-path)]
+              (if handle
+                (do
+                  (utils/verifyPermission handle true)
+                  (p/let [file-handle (.getFileHandle ^js handle basename #js {:create true})
+                          _ (idb/set-item! basename-handle-path file-handle)]
+                    (utils/writeFile file-handle content)))
+                (println "Error: directory handle not exists: " handle-path)))
+            (p/catch (fn [error]
+                       (println "Write local file failed: " {:path path})
+                       (js/console.error error)))))))
+
+     js/window.pfs
+     (js/window.pfs.writeFile (str dir "/" path) content)
+
+     :else
+     nil)))
+
+(defn rename
+  [old-path new-path]
+  (cond
+    (local-db? old-path)
+    ;; create new file
+    ;; delete old file
+    (p/let [[dir basename] (util/get-dir-and-basename old-path)
+            [_ new-basename] (util/get-dir-and-basename new-path)
+            handle (idb/get-item (str "handle" old-path))
+            file (.getFile handle)
+            content (.text file)
+            _ (write-file dir new-basename content)]
+      (unlink old-path nil))
+
+    :else
+    (js/window.pfs.rename old-path new-path)))
 
 (defn stat
   [dir path]
-  (js/window.pfs.stat (str dir "/" path)))
+  (let [append-path (if path
+                      (str "/"
+                           (if (= \/ (first path))
+                             (subs path 1)
+                             path))
+                      "")]
+    (cond
+      (local-db? dir)
+      (if-let [file (get-nfs-file-handle (str "handle/"
+                                              (string/replace-first dir "/" "")
+                                              append-path))]
+        (p/let [file (.getFile file)]
+          (let [get-attr #(gobj/get file %)]
+            {:file/last-modified-at (get-attr "lastModified")
+             :file/size (get-attr "size")
+             :file/type (get-attr "type")}))
+        (p/rejected "File not exists"))
+
+      :else
+      (do
+        (js/window.pfs.stat (str dir append-path))))))
+
+(defn mkdir-if-not-exists
+  [dir]
+  (when dir
+    (let [local? (config/local-db? dir)]
+      (when (or local? js/window.pfs)
+        (util/p-handle
+         (stat dir nil)
+         (fn [_stat])
+         (fn [error]
+           (mkdir dir)))))))
 
 (defn create-if-not-exists
   ([dir path]
@@ -70,5 +276,9 @@
    (fn [_stat] true)
    (fn [_e] false)))
 
-(comment
-  (def dir "/notes"))
+(defn check-directory-permission!
+  [repo]
+  (when (config/local-db? repo)
+    (p/let [handle (idb/get-item (str "handle/" repo))]
+      (when handle
+        (utils/verifyPermission handle true)))))

src/main/frontend/handler.cljs

@@ -13,11 +13,14 @@
             [frontend.handler.page :as page-handler]
             [frontend.handler.repo :as repo-handler]
             [frontend.handler.file :as file-handler]
+            [frontend.handler.editor :as editor-handler]
             [frontend.handler.ui :as ui-handler]
             [frontend.handler.export :as export-handler]
+            [frontend.handler.web.nfs :as nfs]
             [frontend.ui :as ui]
             [goog.object :as gobj]
             [frontend.helper :as helper]
+            [frontend.idb :as idb]
             [lambdaisland.glogi :as log]))
 
 (defn- watch-for-date!
@@ -25,7 +28,8 @@
   (js/setInterval (fn []
                     (state/set-today! (date/today))
                     (when-let [repo (state/get-current-repo)]
-                      (when (db/cloned? repo)
+                      (when (or (db/cloned? repo)
+                                (config/local-db? repo))
                         (let [today-page (string/lower-case (date/today))]
                           (when (empty? (db/get-page-blocks-no-cache repo today-page))
                             (repo-handler/create-today-journal-if-not-exists repo))))))
@@ -55,15 +59,13 @@
 
 (defn clear-stores-and-refresh!
   []
-  (p/let [_ (db/clear-local-storage-and-idb!)]
+  (p/let [_ (idb/clear-local-storage-and-idb!)]
     (let [{:keys [me logged? repos]} (get-me-and-repos)]
       (js/window.location.reload))))
 
 (defn restore-and-setup!
   [me repos logged?]
-  ;; wait until pfs is loaded
-  (let [pfs-loaded? (atom js/window.pfs)
-        interval (atom nil)
+  (let [interval (atom nil)
         inner-fn (fn []
                    (when (and @interval js/window.pfs)
                      (js/clearInterval @interval)
@@ -74,11 +76,15 @@
                                                (ui-handler/add-style-if-exists!))))
                          (p/then
                           (fn []
-                            (if (and (not logged?)
-                                     (not (seq (db/get-files config/local-repo))))
+                            (cond
+                              (and (not logged?)
+                                   (not (seq (db/get-files config/local-repo)))
+                                   ;; Not native local directory
+                                   (not (some config/local-db? (map :url repos))))
                               (repo-handler/setup-local-repo-if-not-exists!)
-                              (state/set-db-restoring! false))
 
+                              :else
+                              (state/set-db-restoring! false))
                             (if (schema-changed?)
                               (do
                                 (notification/show!
@@ -86,18 +92,20 @@
                                  :warning
                                  false)
                                 (let [export-repos (for [repo repos]
-                                                    (when-let [url (:url repo)]
-                                                      (println "Export repo: " url)
-                                                      (export-handler/export-repo-as-zip! url)))]
-                                 (-> (p/all export-repos)
-                                     (p/then (fn []
-                                               (store-schema!)
-                                               (js/setTimeout clear-stores-and-refresh! 5000)))
-                                     (p/catch (fn [error]
-                                                (log/error :export/zip {:error error
-                                                                        :repos repos}))))))
+                                                     (when-let [url (:url repo)]
+                                                       (println "Export repo: " url)
+                                                       (export-handler/export-repo-as-zip! url)))]
+                                  (-> (p/all export-repos)
+                                      (p/then (fn []
+                                                (store-schema!)
+                                                (js/setTimeout clear-stores-and-refresh! 5000)))
+                                      (p/catch (fn [error]
+                                                 (log/error :export/zip {:error error
+                                                                         :repos repos}))))))
                               (store-schema!))
 
+                            (nfs/ask-permission-if-local?)
+
                             (page-handler/init-commands!)
                             (if (seq (:repos me))
                               ;; FIXME: handle error
@@ -107,52 +115,13 @@
                                (fn []
                                  (js/console.error "Failed to request GitHub app tokens."))))
 
-                            (watch-for-date!))))))]
+                            (watch-for-date!)))
+                         (p/catch (fn [error]
+                                    (log/error :db/restore-failed error))))))]
     ;; clear this interval
     (let [interval-id (js/setInterval inner-fn 50)]
       (reset! interval interval-id))))
 
-(defn persist-repo-to-indexeddb!
-  ([]
-   (persist-repo-to-indexeddb! false))
-  ([force?]
-   (let [status (state/get-repo-persist-status)]
-     (doseq [[repo {:keys [last-stored-at last-modified-at] :as repo-status}] status]
-       (when (and (> last-modified-at last-stored-at)
-                  (or force?
-                      (and (state/get-edit-input-id)
-                           (> (- (util/time-ms) last-stored-at) (* 5 60 1000)) ; 5 minutes
-)
-                      (nil? (state/get-edit-input-id))))
-         (p/let [_ (repo-handler/persist-repo! repo)]
-           (state/update-repo-last-stored-at! repo)))))))
-
-(defn periodically-persist-repo-to-indexeddb!
-  []
-  (js/setInterval persist-repo-to-indexeddb! (* 5 1000)))
-
-(defn set-save-before-unload! []
-  (.addEventListener js/window "beforeunload"
-                     (fn [e]
-                       (when (and (not config/dev?) (state/repos-need-to-be-stored?))
-                         (let [notification-id (atom nil)]
-                           (let [id (notification/show!
-                                     [:div
-                                      [:p "It seems that you have some unsaved changes!"]
-                                      (ui/button "Save"
-                                                 :on-click (fn [e]
-                                                             (persist-repo-to-indexeddb!)
-                                                             (notification/show!
-                                                              "Saved successfully!"
-                                                              :success)
-                                                             (and @notification-id (notification/clear! @notification-id))))]
-                                     :warning
-                                     false)]
-                             (reset! notification-id id)))
-                         (let [message "\\o/"]
-                           (set! (.-returnValue (or e js/window.event)) message)
-                           message)))))
-
 (defn- handle-connection-change
   [e]
   (let [online? (= (gobj/get e "type") "online")]
@@ -177,9 +146,22 @@
        (notification/show! "Sorry, it seems that your browser doesn't support IndexedDB, we recommend to use latest Chrome(Chromium) or Firefox(Non-private mode)." :error false)
        (state/set-indexedb-support! false)))
 
-    (restore-and-setup! me repos logged?)
-
-    (periodically-persist-repo-to-indexeddb!)
-
-    (db/run-batch-txs!))
-  (set-save-before-unload!))
+    (p/let [nfs-dbs (idb/get-nfs-dbs)
+            nfs-dbs (map (fn [db]
+                           {:url db :nfs? true}) nfs-dbs)]
+      (let [repos (cond
+                    logged?
+                    (concat
+                     nfs-dbs
+                     (:repos me))
+
+                    (seq nfs-dbs)
+                    nfs-dbs
+
+                    :else
+                    [{:url config/local-repo
+                      :example? true}])]
+        (state/set-repos! repos)
+        (restore-and-setup! me repos logged?)))
+    (db/run-batch-txs!)
+    (editor-handler/periodically-save!)))

src/main/frontend/handler/common.cljs

@@ -18,8 +18,6 @@
     ;; TODO: what if the remote is not named "origin", check the api from isomorphic-git
     (git/resolve-ref repo-url (str "refs/remotes/origin/" branch))))
 
-
-;; Should include un-pushed committed files too
 (defn check-changed-files-status
   ([]
    (check-changed-files-status (state/get-current-repo)))

src/main/frontend/handler/editor.cljs

@@ -5,6 +5,7 @@
             [frontend.handler.git :as git-handler]
             [frontend.handler.ui :as ui-handler]
             [frontend.handler.repo :as repo-handler]
+            [frontend.handler.file :as file-handler]
             [frontend.handler.notification :as notification]
             [frontend.handler.draw :as draw]
             [frontend.handler.expand :as expand]
@@ -614,6 +615,7 @@
                                  (block/parse-block (assoc block :block/content new-value) format))
                                parse-result)
                              after-blocks (rebuild-after-blocks repo file (:end-pos meta) end-pos)
+                             files [[file-path new-content]]
                              transact-fn (fn []
                                            (repo-handler/transact-react-and-alter-file!
                                             repo
@@ -624,7 +626,7 @@
                                              after-blocks)
                                             {:key :block/insert
                                              :data (map (fn [block] (assoc block :block/page page)) blocks)}
-                                            [[file-path new-content]])
+                                            files)
                                            (state/set-editor-op! nil))]
 
                          ;; Replace with batch transactions
@@ -1320,17 +1322,38 @@
               nil)
             (state/conj-selection-block! element up?)))))))
 
+(defn save-block-aux!
+  [block value format]
+  (let [value (text/remove-level-spaces value format true)
+        new-value (block/with-levels value format block)
+        properties (with-timetracking-properties block value)]
+    ;; FIXME: somehow frontend.components.editor's will-unmount event will loop forever
+    ;; maybe we shouldn't save the block/file in "will-unmount" event?
+    (save-block-if-changed! block new-value
+                            {:custom-properties properties})))
+
 (defn save-block!
   [{:keys [format block id repo dummy?] :as state} value]
   (when (or (:db/id (db/entity repo [:block/uuid (:block/uuid block)]))
             dummy?)
-    (let [value (text/remove-level-spaces value format true)
-          new-value (block/with-levels value format block)
-          properties (with-timetracking-properties block value)]
-      ;; FIXME: somehow frontend.components.editor's will-unmount event will loop forever
-      ;; maybe we shouldn't save the block/file in "will-unmount" event?
-      (save-block-if-changed! block new-value
-                              {:custom-properties properties}))))
+    (save-block-aux! block value format)))
+
+(defn save-current-block-when-idle!
+  []
+  (when-let [repo (state/get-current-repo)]
+    (when (state/input-idle? repo)
+      (let [input-id (state/get-edit-input-id)
+            block (state/get-edit-block)
+            elem (and input-id (gdom/getElement input-id))
+            db-block (db/entity [:block/uuid (:block/uuid block)])
+            db-content (:block/content db-block)
+            db-content-without-heading (and db-content
+                                            (util/safe-subs db-content (:block/level db-block)))
+            value (and elem (gobj/get elem "value"))]
+        (when (and block value db-content-without-heading
+                   (not= (string/trim db-content-without-heading)
+                         (string/trim value)))
+          (save-block-aux! block value (:block/format block)))))))
 
 (defn on-up-down
   [state e up?]
@@ -1957,3 +1980,14 @@
           (state/set-editor-show-block-search! false)
           (state/set-editor-show-page-search! false)
           (state/set-editor-show-page-search-hashtag! false))))))
+
+(defn periodically-save!
+  []
+  (js/setInterval save-current-block-when-idle! 3000))
+
+(defn get-current-input-value
+  []
+  (let [edit-input-id (state/get-edit-input-id)
+        input (and edit-input-id (gdom/getElement edit-input-id))]
+    (when input
+      (gobj/get input "value"))))

src/main/frontend/handler/file.cljs

@@ -15,7 +15,9 @@
             [frontend.format :as format]
             [clojure.string :as string]
             [frontend.history :as history]
-            [frontend.handler.project :as project-handler]))
+            [frontend.handler.project :as project-handler]
+            [lambdaisland.glogi :as log]
+            ["ignore" :as Ignore]))
 
 (defn load-file
   [repo-url path]
@@ -59,11 +61,11 @@
                (subs path 1)
                path)]
     (some (fn [pattern]
-           (let [pattern (if (and (string? pattern)
-                                  (not= \/ (first pattern)))
-                           (str "/" pattern)
-                           pattern)]
-             (string/starts-with? (str "/" path) pattern))) patterns)))
+            (let [pattern (if (and (string? pattern)
+                                   (not= \/ (first pattern)))
+                            (str "/" pattern)
+                            pattern)]
+              (string/starts-with? (str "/" path) pattern))) patterns)))
 
 (defn restore-config!
   ([repo-url project-changed-check?]
@@ -71,7 +73,8 @@
   ([repo-url config-content project-changed-check?]
    (let [old-project (:project (state/get-config))
          new-config (db/reset-config! repo-url config-content)]
-     (when project-changed-check?
+     (when (and (not (config/local-db? repo-url))
+                project-changed-check?)
        (let [new-project (:project new-config)
              project-name (:name old-project)]
          (when-not (= new-project old-project)
@@ -80,7 +83,7 @@
 (defn load-files
   [repo-url]
   (state/set-cloning! false)
-  (state/set-state! :repo/loading-files? true)
+  (state/set-loading-files! true)
   (p/let [files (git/list-files repo-url)
           files (bean/->clj files)
           config-content (load-file repo-url (str config/app-name "/" config/config-file))
@@ -98,15 +101,17 @@
         files (only-text-formats files)]
     (-> (p/all (load-multiple-files repo-url files))
         (p/then (fn [contents]
-                  (ok-handler
-                   (cond->
-                    (zipmap files contents)
-
-                     (seq images)
-                     (merge (zipmap images (repeat (count images) "")))))))
+                  (let [file-contents (cond->
+                                       (zipmap files contents)
+
+                                        (seq images)
+                                        (merge (zipmap images (repeat (count images) ""))))
+                        file-contents (for [[file content] file-contents]
+                                        {:file/path file
+                                         :file/content content})]
+                    (ok-handler file-contents))))
         (p/catch (fn [error]
-                   (println "load files failed: ")
-                   (js/console.dir error))))))
+                   (log/error :load-files-error error))))))
 
 (defn alter-file
   [repo path content {:keys [reset? re-render-root? add-history? update-status?]
@@ -124,7 +129,7 @@
         (db/reset-file! repo path content))
       (db/set-file-content! repo path content))
     (util/p-handle
-     (fs/write-file (util/get-repo-dir repo) path content)
+     (fs/write-file (util/get-repo-dir repo) path content original-content)
      (fn [_]
        (git-handler/git-add repo path update-status?)
        (when (= path (str config/app-name "/" config/config-file))
@@ -148,7 +153,7 @@
                                                 :re-render-root? false
                                                 :update-status? true})]
          (route-handler/redirect! {:to :file
-                                   :path-params {:path path}})))))  )
+                                   :path-params {:path path}}))))))
 
 (defn alter-files
   ([repo files]
@@ -157,42 +162,46 @@
                 :or {add-history? true
                      update-status? true
                      reset? false}}]
-   (let [files-tx (mapv (fn [[path content]]
-                          (let [original-content (db/get-file-no-sub repo path)]
-                            [path original-content content])) files)
-         write-file-f (fn [[path content]]
-                        (if reset?
-                          (db/reset-file! repo path content)
-                          (db/set-file-content! repo path content))
-                        (util/p-handle
-                         (fs/write-file (util/get-repo-dir repo) path content)
-                         (fn [_])
-                         (fn [error]
-                           (println "Write file failed, path: " path ", content: " content)
-                           (js/console.error error))))
-         git-add-f (fn [_result]
-                     (let [add-helper
-                           (fn []
-                             (doall
-                              (map
-                               (fn [[path content]]
-                                 (git-handler/git-add repo path update-status?))
-                               files)))]
-                       (-> (p/all (add-helper))
-                           (p/then (fn [_]
-                                     (when git-add-cb
-                                       (git-add-cb))))
-                           (p/catch (fn [error]
-                                      (println "Git add failed:")
-                                      (js/console.error error)))))
-                     (ui-handler/re-render-file!)
-                     (when add-history?
-                       (history/add-history! repo files-tx)))]
-     (-> (p/all (doall (map write-file-f files)))
-         (p/then git-add-f)
-         (p/catch (fn [error]
-                    (println "Alter files failed:")
-                    (js/console.error error)))))))
+   (p/let [file->content (let [paths (map first files)]
+                           (zipmap paths
+                                   (map (fn [path] (db/get-file-no-sub repo path)) paths)))]
+     (let [files-tx (mapv (fn [[path content]]
+                            (let [original-content (get file->content path)]
+                              [path original-content content])) files)
+           write-file-f (fn [[path content]]
+                          (if reset?
+                            (db/reset-file! repo path content)
+                            (db/set-file-content! repo path content))
+                          (let [original-content (get file->content path)]
+                            (-> (p/let [_ (fs/check-directory-permission! repo)]
+                                  (fs/write-file (util/get-repo-dir repo) path content original-content))
+                                (p/catch (fn [error]
+                                           (log/error :write-file/failed {:path path
+                                                                          :content content
+                                                                          :error error}))))))
+           git-add-f (fn [_result]
+                       (let [add-helper
+                             (fn []
+                               (doall
+                                (map
+                                 (fn [[path content]]
+                                   (git-handler/git-add repo path update-status?))
+                                 files)))]
+                         (-> (p/all (add-helper))
+                             (p/then (fn [_]
+                                       (when git-add-cb
+                                         (git-add-cb))))
+                             (p/catch (fn [error]
+                                        (println "Git add failed:")
+                                        (js/console.error error)))))
+                       (ui-handler/re-render-file!)
+                       (when add-history?
+                         (history/add-history! repo files-tx)))]
+       (-> (p/all (doall (map write-file-f files)))
+           (p/then git-add-f)
+           (p/catch (fn [error]
+                      (println "Alter files failed:")
+                      (js/console.error error))))))))
 
 (defn remove-file!
   [repo file]
@@ -222,3 +231,10 @@
     (let [path (:file/path file)
           content (db/get-file path)]
       (alter-file repo path content {:re-render-root? true}))))
+
+(defn ignore-files
+  [pattern paths]
+  (-> (Ignore)
+      (.add pattern)
+      (.filter (bean/->js paths))
+      (bean/->clj)))

src/main/frontend/handler/git.cljs

@@ -1,5 +1,4 @@
 (ns frontend.handler.git
-  (:refer-clojure :exclude [clone load-file])
   (:require [frontend.util :as util :refer-macros [profile]]
             [promesa.core :as p]
             [frontend.state :as state]
@@ -10,6 +9,7 @@
             [frontend.handler.notification :as notification]
             [frontend.handler.route :as route-handler]
             [frontend.handler.common :as common-handler]
+            [frontend.config :as config]
             [cljs-time.local :as tl]
             [frontend.helper :as helper]))
 
@@ -31,12 +31,13 @@
   ([repo-url file]
    (git-add repo-url file true))
   ([repo-url file update-status?]
-   (-> (p/let [result (git/add repo-url file)]
-         (when update-status?
-           (common-handler/check-changed-files-status)))
-       (p/catch (fn [error]
-                  (println "git add '" file "' failed: " error)
-                  (js/console.error error))))))
+   (when-not (config/local-db? repo-url)
+     (-> (p/let [result (git/add repo-url file)]
+           (when update-status?
+             (common-handler/check-changed-files-status)))
+         (p/catch (fn [error]
+                    (println "git add '" file "' failed: " error)
+                    (js/console.error error)))))))
 
 (defn commit-and-force-push!
   [commit-message pushing?]

src/main/frontend/handler/image.cljs

@@ -33,8 +33,9 @@
                      (subs path 1)
                      path)]
           (util/p-handle
-           (fs/read-file-2 (util/get-repo-dir (state/get-current-repo))
-                           path)
+           (fs/read-file (util/get-repo-dir (state/get-current-repo))
+                           path
+                           {})
            (fn [blob]
              (let [blob (js/Blob. (array blob) (clj->js {:type "image"}))
                    img-url (image/create-object-url blob)]

src/main/frontend/handler/project.cljs

@@ -16,38 +16,41 @@
   ([ok-handler]
    (create-project! (state/get-current-project) ok-handler))
   ([project ok-handler]
-   (let [config (state/get-config)
-         data {:name project
-               :repo (state/get-current-repo)
-               :settings (or (get config :project)
-                             {:name project})}]
-     (util/post (str config/api "projects")
-                data
-                (fn [result]
-                  (when-not (:message result) ; exists
-                    (swap! state/state
-                           update-in [:me :projects]
-                           (fn [projects]
-                             (util/distinct-by :name (conj projects result))))
-                    (ok-handler project)))
-                (fn [error]
-                  (js/console.dir error)
-                  (notification/show! (util/format "Project \"%s\" already taken, please change to another name." project) :error))))))
+   (when (state/logged?)
+     (let [config (state/get-config)
+           data {:name project
+                 :repo (state/get-current-repo)
+                 :settings (or (get config :project)
+                               {:name project})}]
+       (util/post (str config/api "projects")
+                  data
+                  (fn [result]
+                    (when-not (:message result) ; exists
+                      (swap! state/state
+                             update-in [:me :projects]
+                             (fn [projects]
+                               (util/distinct-by :name (conj projects result))))
+                      (ok-handler project)))
+                  (fn [error]
+                    (js/console.dir error)
+                    (notification/show! (util/format "Project \"%s\" already taken, please change to another name." project) :error)))))))
 
 (defn exists-or-create!
   [ok-handler modal-content]
-  (if-let [project (state/get-current-project)]
-    (if (project-exists? project)
-      (ok-handler project)
-      (create-project! ok-handler))
-    (state/set-modal! modal-content)))
+  (when (state/logged?)
+    (if-let [project (state/get-current-project)]
+      (if (project-exists? project)
+        (ok-handler project)
+        (create-project! ok-handler))
+      (state/set-modal! modal-content))))
 
 (defn add-project!
   [project]
-  (create-project! project
-                   (fn []
-                     (notification/show! (util/format "Project \"%s\" was created successfully." project) :success)
-                     (state/close-modal!))))
+  (when (state/logged?)
+    (create-project! project
+                     (fn []
+                       (notification/show! (util/format "Project \"%s\" was created successfully." project) :success)
+                       (state/close-modal!)))))
 
 (defn sync-project-settings!
   ([]
@@ -55,18 +58,19 @@
      (let [settings (:project (state/get-config))]
        (sync-project-settings! project-name settings))))
   ([project-name settings]
-   (when-let [repo (state/get-current-repo)]
-     (if (project-exists? project-name)
-       (util/post (str config/api "projects/" project-name)
-                  {:name project-name
-                   :settings settings
-                   :repo repo}
-                  (fn [response]
-                    (notification/show! "Project settings changed successfully!" :success))
-                  (fn [error]
-                    (println "Project settings updated failed, reason: ")
-                    (js/console.dir error)))
-       (when (and settings
-                  (not (string/blank? (:name settings)))
-                  (>= (count (string/trim (:name settings))) 2))
-         (add-project! (:name settings)))))))
+   (when (state/logged?)
+     (when-let [repo (state/get-current-repo)]
+       (if (project-exists? project-name)
+         (util/post (str config/api "projects/" project-name)
+                    {:name project-name
+                     :settings settings
+                     :repo repo}
+                    (fn [response]
+                      (notification/show! "Project settings changed successfully!" :success))
+                    (fn [error]
+                      (println "Project settings updated failed, reason: ")
+                      (js/console.dir error)))
+         (when (and settings
+                    (not (string/blank? (:name settings)))
+                    (>= (count (string/trim (:name settings))) 2))
+           (add-project! (:name settings))))))))

src/main/frontend/handler/repo.cljs

@@ -67,10 +67,12 @@
   (spec/validate :repos/url repo-url)
   (let [repo-dir (util/get-repo-dir repo-url)
         format (state/get-preferred-format)
-        path (str "pages/contents." (config/get-file-extension format))
+        path (str (state/get-pages-directory)
+                  "/contents."
+                  (config/get-file-extension format))
         file-path (str "/" path)
         default-content (util/default-content-with-title format "contents")]
-    (p/let [_ (fs/mkdir-if-not-exists (str repo-dir "/pages"))
+    (p/let [_ (fs/mkdir-if-not-exists (str repo-dir "/" (state/get-pages-directory)))
             file-exists? (fs/create-if-not-exists repo-dir file-path default-content)]
       (when-not file-exists?
         (db/reset-file! repo-url path default-content)
@@ -104,6 +106,8 @@
    (create-today-journal-if-not-exists repo-url nil))
   ([repo-url content]
    (spec/validate :repos/url repo-url)
+   (when (config/local-db? repo-url)
+     (fs/check-directory-permission! repo-url))
    (let [repo-dir (util/get-repo-dir repo-url)
          format (state/get-preferred-format repo-url)
          title (date/today)
@@ -139,49 +143,60 @@
 (defn create-default-files!
   [repo-url]
   (spec/validate :repos/url repo-url)
-  (when (state/logged?)
-    (create-config-file-if-not-exists repo-url)
-    (create-today-journal-if-not-exists repo-url)
-    (create-contents-file repo-url)
-    (create-custom-theme repo-url)))
+  (create-config-file-if-not-exists repo-url)
+  (create-today-journal-if-not-exists repo-url)
+  (create-contents-file repo-url)
+  (create-custom-theme repo-url))
+
+(defn- parse-files-and-load-to-db!
+  [repo-url files {:keys [first-clone? delete-files delete-blocks re-render? re-render-opts] :as opts}]
+  (state/set-loading-files! false)
+  (state/set-importing-to-db! true)
+  (let [file-paths (map :file/path files)
+        parsed-files (filter
+                      (fn [file]
+                        (let [format (format/get-format (:file/path file))]
+                          (contains? config/mldoc-support-formats format)))
+                      files)
+        blocks-pages (if (seq parsed-files)
+                       (db/extract-all-blocks-pages repo-url parsed-files)
+                       [])]
+    (db/reset-contents-and-blocks! repo-url files blocks-pages delete-files delete-blocks)
+    (let [config-file (str config/app-name "/" config/config-file)]
+      (if (contains? (set file-paths) config-file)
+        (when-let [content (some #(when (= (:file/path %) config-file)
+                                    (:file/content %)) files)]
+          (file-handler/restore-config! repo-url content true))))
+    (when first-clone? (create-default-files! repo-url))
+    (when re-render?
+      (ui-handler/re-render-root! re-render-opts))
+    (state/set-importing-to-db! false)))
 
 (defn load-repo-to-db!
-  [repo-url diffs first-clone?]
+  [repo-url {:keys [first-clone? diffs nfs-files]}]
   (spec/validate :repos/url repo-url)
-  (let [load-contents (fn [files delete-files delete-blocks re-render?]
+  (let [load-contents (fn [files option]
                         (file-handler/load-files-contents!
                          repo-url
                          files
-                         (fn [contents]
-                           (state/set-state! :repo/loading-files? false)
-                           (state/set-state! :repo/importing-to-db? true)
-                           (let [parsed-files (filter
-                                               (fn [[file _]]
-                                                 (let [format (format/get-format file)]
-                                                   (contains? config/mldoc-support-formats format)))
-                                               contents)
-                                 blocks-pages (if (seq parsed-files)
-                                                (db/extract-all-blocks-pages repo-url parsed-files)
-                                                [])]
-                             (db/reset-contents-and-blocks! repo-url contents blocks-pages delete-files delete-blocks)
-                             (let [config-file (str config/app-name "/" config/config-file)]
-                               (if (contains? (set files) config-file)
-                                 (when-let [content (get contents config-file)]
-                                   (file-handler/restore-config! repo-url content true))))
-                             (when first-clone? (create-default-files! repo-url))
-                             (state/set-state! :repo/importing-to-db? false)
-                             (when re-render?
-                               (ui-handler/re-render-root!))))))]
-    (if first-clone?
+                         (fn [files-contents]
+                           (parse-files-and-load-to-db! repo-url files-contents option))))]
+    (cond
+      (and (not (seq diffs)) (seq nfs-files))
+      (parse-files-and-load-to-db! repo-url nfs-files {:first-clone? true})
+
+      first-clone?
       (->
        (p/let [files (file-handler/load-files repo-url)]
-         (load-contents files nil nil false))
+         (load-contents files {:first-clone? first-clone?}))
        (p/catch (fn [error]
                   (println "loading files failed: ")
                   (js/console.dir error)
                   ;; Empty repo
                   (create-default-files! repo-url)
-                  (state/set-state! :repo/loading-files? false))))
+                  (state/set-loading-files! false))))
+
+      :else
       (when (seq diffs)
         (let [filter-diffs (fn [type] (->> (filter (fn [f] (= type (:type f))) diffs)
                                            (map :path)))
@@ -194,22 +209,24 @@
               delete-pages (if (seq remove-files)
                              (db/delete-pages-by-files remove-files)
                              [])
-              add-or-modify-files (util/remove-nils (concat add-files modify-files))]
-          (load-contents add-or-modify-files (concat delete-files delete-pages) delete-blocks true))))))
-
-(defn persist-repo!
-  [repo]
-  (spec/validate :repos/url repo)
-  (when-let [files-conn (db/get-files-conn repo)]
-    (db/persist repo @files-conn true))
-  (when-let [db (db/get-conn repo)]
-    (db/persist repo db false)))
+              add-or-modify-files (some->>
+                                   (concat modify-files add-files)
+                                   (util/remove-nils))
+              options {:first-clone? first-clone?
+                       :delete-files (concat delete-files delete-pages)
+                       :delete-blocks delete-blocks
+                       :re-render? true}]
+          (if (seq nfs-files)
+            (parse-files-and-load-to-db! repo-url nfs-files
+                                         (assoc options :re-render-opts {:clear-all-query-state? true}))
+            (load-contents add-or-modify-files options)))))))
 
 (defn load-db-and-journals!
   [repo-url diffs first-clone?]
   (spec/validate :repos/url repo-url)
   (when (or diffs first-clone?)
-    (load-repo-to-db! repo-url diffs first-clone?)))
+    (load-repo-to-db! repo-url {:first-clone? first-clone?
+                                :diffs diffs})))
 
 (defn transact-react-and-alter-file!
   [repo tx transact-option files]
@@ -224,27 +241,9 @@
     (when (seq pages)
       (let [children-tx (mapcat #(db/rebuild-page-blocks-children repo %) pages)]
         (when (seq children-tx)
-          (db/transact! repo children-tx)))))
-  (when (seq files)
-    (file-handler/alter-files repo files)))
-
-; FIXME: Unused
-(defn persist-repo-metadata!
-  [repo]
-  (spec/validate :repos/url repo)
-  (let [files (db/get-files repo)]
+          (db/transact! repo children-tx))))
     (when (seq files)
-      (let [data (db/get-sync-metadata repo)
-            data-str (pr-str data)]
-        (file-handler/alter-file repo
-                                 (str config/app-name "/" config/metadata-file)
-                                 data-str
-                                 {:reset? false})))))
-
-(defn periodically-persist-app-metadata
-  [repo-url]
-  (js/setInterval #(persist-repo-metadata! repo-url)
-                  (* 5 60 1000)))
+      (file-handler/alter-files repo files))))
 
 (declare push)
 
@@ -351,7 +350,7 @@
   (let [status (db/get-key-value repo-url :git/status)]
     (if (and
          (db/cloned? repo-url)
-         (not (state/get-edit-input-id))
+         (state/input-idle? repo-url)
          (or (not= status :pushing)
              custom-commit?))
       (-> (p/let [files (js/window.workerThread.getChangedFiles (util/get-repo-dir (state/get-current-repo)))]
@@ -445,15 +444,26 @@
 (defn remove-repo!
   [{:keys [id url] :as repo}]
   (spec/validate :repos/repo repo)
-  (util/delete (str config/api "repos/" id)
-               (fn []
-                 (db/remove-conn! url)
-                 (db/remove-db! url)
-                 (db/remove-files-db! url)
-                 (fs/rmdir (util/get-repo-dir url))
-                 (state/delete-repo! repo))
-               (fn [error]
-                 (prn "Delete repo failed, error: " error))))
+  (let [delete-db-f (fn []
+                      (db/remove-conn! url)
+                      (db/remove-db! url)
+                      (db/remove-files-db! url)
+                      (fs/rmdir (util/get-repo-dir url))
+                      (state/delete-repo! repo))]
+    (if (config/local-db? url)
+      (do
+        (delete-db-f)
+        ;; clear handles
+)
+      (util/delete (str config/api "repos/" id)
+                   delete-db-f
+                   (fn [error]
+                     (prn "Delete repo failed, error: " error))))))
+
+(defn start-repo-db-if-not-exists!
+  [repo option]
+  (state/set-current-repo! repo)
+  (db/start-db-conn! nil repo option))
 
 (defn setup-local-repo-if-not-exists!
   []

src/main/frontend/handler/route.cljs

@@ -79,7 +79,9 @@
   (let [{:keys [data path-params]} route
         title (get-title (:name data) path-params)]
     (util/set-title! title)
-    (ui-handler/scroll-and-highlight! nil)))
+    (if-let [fragment (util/get-fragment)]
+      (ui-handler/highlight-element! fragment)
+      (util/scroll-to-top))))
 
 (defn go-to-search!
   []

src/main/frontend/handler/ui.cljs

@@ -30,12 +30,17 @@
 
 
 (defn re-render-root!
-  []
-  (when-let [component (state/get-root-component)]
-    (db/clear-query-state-without-refs-and-embeds!)
-    (rum/request-render component)
-    (doseq [component (state/get-custom-query-components)]
-      (rum/request-render component))))
+  ([]
+   (re-render-root! {}))
+  ([{:keys [clear-all-query-state?]
+     :or {clear-all-query-state? false}}]
+   (when-let [component (state/get-root-component)]
+     (if clear-all-query-state?
+       (db/clear-query-state!)
+       (db/clear-query-state-without-refs-and-embeds!))
+     (rum/request-render component)
+     (doseq [component (state/get-custom-query-components)]
+       (rum/request-render component)))))
 
 (defn re-render-file!
   []
@@ -65,8 +70,7 @@
 (defn scroll-and-highlight!
   [state]
   (if-let [fragment (util/get-fragment)]
-    (highlight-element! fragment)
-    (util/scroll-to-top))
+    (highlight-element! fragment))
   state)
 
 (defn add-style-if-exists!

src/main/frontend/handler/user.cljs

@@ -2,6 +2,7 @@
   (:require [frontend.util :as util :refer-macros [profile]]
             [frontend.state :as state]
             [frontend.db :as db]
+            [frontend.idb :as idb]
             [frontend.config :as config]
             [frontend.storage :as storage]
             [promesa.core :as p]
@@ -61,7 +62,7 @@
 (defn sign-out!
   [e]
   (->
-   (db/clear-local-storage-and-idb!)
+   (idb/clear-local-storage-and-idb!)
    (p/catch (fn [e]
               (println "sign out error: ")
               (js/console.dir e)))

src/main/frontend/handler/web/nfs.cljs

@@ -0,0 +1,235 @@
+(ns frontend.handler.web.nfs
+  "The File System Access API, https://web.dev/file-system-access/."
+  (:require [cljs-bean.core :as bean]
+            [promesa.core :as p]
+            [goog.object :as gobj]
+            [goog.dom :as gdom]
+            [frontend.util :as util]
+            ["/frontend/utils" :as utils]
+            [frontend.handler.repo :as repo-handler]
+            [frontend.handler.file :as file-handler]
+            [frontend.idb :as idb]
+            [frontend.state :as state]
+            [clojure.string :as string]
+            [clojure.set :as set]
+            [frontend.ui :as ui]
+            [frontend.fs :as fs]
+            [frontend.db :as db]
+            [frontend.config :as config]
+            [lambdaisland.glogi :as log]))
+
+(defn remove-ignore-files
+  [files]
+  (let [files (remove (fn [f]
+                        (string/starts-with? (:file/path f) ".git/"))
+                      files)]
+    (if-let [ignore-file (some #(when (= (:file/name %) ".gitignore")
+                                  %) files)]
+      (if-let [file (:file/file ignore-file)]
+        (p/let [content (.text file)]
+          (when content
+            (let [paths (set (file-handler/ignore-files content (map :file/path files)))]
+              (when (seq paths)
+                (filter (fn [f] (contains? paths (:file/path f))) files)))))
+        (p/resolved files))
+      (p/resolved files))))
+
+(defn- ->db-files
+  [dir-name result]
+  (let [result (flatten (bean/->clj result))]
+    (map (fn [file]
+           (let [handle (gobj/get file "handle")
+                 get-attr #(gobj/get file %)
+                 path (-> (get-attr "webkitRelativePath")
+                          (string/replace-first (str dir-name "/") ""))]
+             {:file/name (get-attr "name")
+              :file/path path
+              :file/last-modified-at (get-attr "lastModified")
+              :file/size (get-attr "size")
+              :file/type (get-attr "type")
+              :file/file file
+              :file/handle handle})) result)))
+
+(defn- filter-markup-and-built-in-files
+  [files]
+  (filter (fn [file]
+            (contains? (set/union config/markup-formats #{:css :edn})
+                       (keyword (util/get-file-ext (:file/path file)))))
+          files))
+
+(defn- set-files!
+  [handles]
+  (doseq [[path handle] handles]
+    (let [handle-path (str config/local-handle-prefix path)]
+      (idb/set-item! handle-path handle)
+      (fs/add-nfs-file-handle! handle-path handle))))
+
+(defn ls-dir-files
+  []
+  (let [path-handles (atom {})]
+    (->
+     (p/let [result (utils/openDirectory #js {:recursive true}
+                                         (fn [path handle]
+                                           (swap! path-handles assoc path handle)))
+             _ (state/set-loading-files! true)
+             root-handle (nth result 0)
+             dir-name (gobj/get root-handle "name")
+             repo (str config/local-db-prefix dir-name)
+             root-handle-path (str config/local-handle-prefix dir-name)
+             _ (idb/set-item! root-handle-path root-handle)
+             _ (fs/add-nfs-file-handle! root-handle-path root-handle)
+             result (nth result 1)
+             files (-> (->db-files dir-name result)
+                       remove-ignore-files)
+             _ (let [file-paths (set (map :file/path files))]
+                 (swap! path-handles (fn [handles]
+                                       (->> handles
+                                            (filter (fn [[path _handle]]
+                                                      (contains? file-paths
+                                                                 (string/replace-first path (str dir-name "/") ""))))
+                                            (into {})))))
+             _ (set-files! @path-handles)
+             markup-files (filter-markup-and-built-in-files files)]
+       (-> (p/all (map (fn [file]
+                         (p/let [content (.text (:file/file file))]
+                           (assoc file :file/content content))) markup-files))
+           (p/then (fn [result]
+                     _ (state/set-loading-files! false)
+                     (let [files (map #(dissoc % :file/file) result)]
+                       (repo-handler/start-repo-db-if-not-exists! repo {:db-type :local-native-fs})
+                       (repo-handler/load-repo-to-db! repo
+                                                      {:first-clone? true
+                                                       :nfs-files files})
+
+                       (state/add-repo! {:url repo :nfs? true}))))
+           (p/catch (fn [error]
+                      (log/error :nfs/load-files-error error)))))
+     (p/catch (fn [error]
+                (when (not= "AbortError" (gobj/get error "name"))
+                  (log/error :nfs/open-dir-error error)))))))
+
+(defn open-file-picker
+  "Shows a file picker that lets a user select a single existing file, returning a handle for the selected file. "
+  ([]
+   (open-file-picker {}))
+  ([option]
+   (js/window.showOpenFilePicker (bean/->js option))))
+
+(defn get-local-repo
+  []
+  (when-let [repo (state/get-current-repo)]
+    (when (config/local-db? repo)
+      repo)))
+
+(defn ask-permission
+  [repo]
+  (fn [close-fn]
+    [:div
+     [:p.text-gray-700
+      "Grant native filesystem permission for directory: "
+      [:b (config/get-local-dir repo)]]
+     (ui/button
+      "Grant"
+      :on-click (fn []
+                  (fs/check-directory-permission! repo)
+                  (close-fn)))]))
+
+(defn ask-permission-if-local? []
+  (when-let [repo (get-local-repo)]
+    (state/set-modal! (ask-permission repo))))
+
+(defn- compute-diffs
+  [old-files new-files]
+  (let [ks [:file/path :file/last-modified-at]
+        ->set (fn [files ks]
+                (when (seq files)
+                  (->> files
+                       (map #(select-keys % ks))
+                       set)))
+        old-files (->set old-files ks)
+        new-files (->set new-files ks)
+        file-path-set-f (fn [col] (set (map :file/path col)))
+        get-file-f (fn [files path] (some #(when (= (:file/path %) path) %) files))
+        old-file-paths (file-path-set-f old-files)
+        new-file-paths (file-path-set-f new-files)
+        added (set/difference new-file-paths old-file-paths)
+        deleted (set/difference old-file-paths new-file-paths)
+        modified (set/difference new-file-paths added)]
+    {:added added
+     :modified modified
+     :deleted deleted}))
+
+(defn- reload-dir!
+  [repo]
+  (when (and repo (config/local-db? repo))
+    (let [old-files (db/get-files-full repo)
+          dir-name (config/get-local-dir repo)
+          handle-path (str config/local-handle-prefix dir-name)
+          path-handles (atom {})]
+      (state/set-graph-syncing? true)
+      (p/let [handle (idb/get-item handle-path)
+              _ (when handle (utils/verifyPermission handle true))
+              files-result (utils/getFiles handle true
+                                           (fn [path handle]
+                                             (swap! path-handles assoc path handle)))
+              new-files (-> (->db-files dir-name files-result)
+                            remove-ignore-files)
+              _ (let [file-paths (set (map :file/path new-files))]
+                  (swap! path-handles (fn [handles]
+                                        (->> handles
+                                             (filter (fn [[path _handle]]
+                                                       (contains? file-paths
+                                                                  (string/replace-first path (str dir-name "/") ""))))
+                                             (into {})))))
+              _ (set-files! @path-handles)
+              get-file-f (fn [path files] (some #(when (= (:file/path %) path) %) files))
+              {:keys [added modified deleted] :as diffs} (compute-diffs old-files new-files)
+              ;; Use the same labels as isomorphic-git
+              rename-f (fn [typ col] (mapv (fn [file] {:type typ :path file}) col))
+              _ (when (seq deleted)
+                  (p/all (map (fn [path]
+                                (let [handle-path (str handle-path path)]
+                                  (idb/remove-item! handle-path)
+                                  (fs/remove-nfs-file-handle! handle-path))) deleted)))
+              added-or-modified (set (concat added modified))
+              _ (when (seq added-or-modified)
+                  (p/all (map (fn [path]
+                                (when-let [handle (get @path-handles path)]
+                                  (idb/set-item! (str handle-path path) handle))) added-or-modified)))]
+        (-> (p/all (map (fn [path]
+                          (when-let [file (get-file-f path new-files)]
+                            (p/let [content (.text (:file/file file))]
+                              (assoc file :file/content content)))) added-or-modified))
+            (p/then (fn [result]
+                      (let [files (map #(dissoc % :file/file :file/handle) result)
+                            non-modified? (fn [file]
+                                            (let [content (:file/content file)
+                                                  old-content (:file/content (get-file-f (:file/path file) old-files))]
+                                              (= content old-content)))
+                            non-modified-files (->> (filter non-modified? files)
+                                                    (map :file/path))
+                            modified-files (remove non-modified? files)
+                            modified (set/difference (set modified) (set non-modified-files))
+                            diffs (concat
+                                   (rename-f "remove" deleted)
+                                   (rename-f "add" added)
+                                   (rename-f "modify" modified))]
+                        (when (or (and (seq diffs) (seq modified-files))
+                                  (seq diffs) ; delete
+)
+                          (repo-handler/load-repo-to-db! repo
+                                                         {:diffs diffs
+                                                          :nfs-files modified-files})))))
+            (p/catch (fn [error]
+                       (log/error :nfs/load-files-error error)))
+            (p/finally (fn [_]
+                         (state/set-graph-syncing? false))))))))
+
+(defn- refresh!
+  [repo]
+  (when repo
+    (reload-dir! repo)))
+
+(defn supported?
+  []
+  (utils/nfsSupported))

src/main/frontend/helper.cljs

@@ -16,15 +16,15 @@
     (when (or (seq repos)
               (seq installation-ids))
       (util/post (str config/api "refresh_github_token")
-        {:installation-ids installation-ids
-         :repos repos}
-        (fn [result]
-          (state/set-github-installation-tokens! result)
-          (when ok-handler (ok-handler)))
-        (fn [error]
-          (log/error :token/http-request-failed error)
-          (js/console.dir error)
-          (when error-handler (error-handler)))))))
+                 {:installation-ids installation-ids
+                  :repos repos}
+                 (fn [result]
+                   (state/set-github-installation-tokens! result)
+                   (when ok-handler (ok-handler)))
+                 (fn [error]
+                   (log/error :token/http-request-failed error)
+                   (js/console.dir error)
+                   (when error-handler (error-handler)))))))
 
 (defn- get-github-token*
   [repo]
@@ -47,20 +47,19 @@
   ([]
    (get-github-token  (state/get-current-repo)))
   ([repo]
-   (js/Promise.
-     (fn [resolve reject]
-       (let [{:keys [expired? token exist?]} (get-github-token* repo)
-             valid-token? (and exist? (not expired?))]
-        (if valid-token?
-          (resolve token)
-          (request-app-tokens!
-            (fn []
-              (let [{:keys [expired? token exist?] :as token-m} (get-github-token* repo)
-                    valid-token? (and exist? (not expired?))]
-                (if valid-token?
-                  (resolve token)
-                  (do (log/error :token/failed-get-token token-m)
-                      (reject)))))
-            nil)))))))
-
-
+   (when-not (config/local-db? repo)
+     (js/Promise.
+      (fn [resolve reject]
+        (let [{:keys [expired? token exist?]} (get-github-token* repo)
+              valid-token? (and exist? (not expired?))]
+          (if valid-token?
+            (resolve token)
+            (request-app-tokens!
+             (fn []
+               (let [{:keys [expired? token exist?] :as token-m} (get-github-token* repo)
+                     valid-token? (and exist? (not expired?))]
+                 (if valid-token?
+                   (resolve token)
+                   (do (log/error :token/failed-get-token token-m)
+                       (reject)))))
+             nil))))))))

src/main/frontend/idb.cljs

@@ -0,0 +1,52 @@
+(ns frontend.idb
+  (:require ["localforage" :as localforage]
+            [cljs-bean.core :as bean]
+            [goog.object :as gobj]
+            [promesa.core :as p]
+            [clojure.string :as string]
+            [frontend.config :as config]
+            [frontend.storage :as storage]))
+
+;; offline db
+(def store-name "dbs")
+(.config localforage
+         (bean/->js
+          {:name "logseq-datascript"
+           :version 1.0
+           :storeName store-name}))
+
+(defonce localforage-instance (.createInstance localforage store-name))
+
+(defn clear-idb!
+  []
+  (p/let [_ (.clear localforage-instance)
+          dbs (js/window.indexedDB.databases)]
+    (doseq [db dbs]
+      (js/window.indexedDB.deleteDatabase (gobj/get db "name")))))
+
+(defn clear-local-storage-and-idb!
+  []
+  (storage/clear)
+  (clear-idb!))
+
+(defn remove-item!
+  [key]
+  (.removeItem localforage-instance key))
+
+(defn set-item!
+  [key value]
+  (.setItem localforage-instance key value))
+
+(defn get-item
+  [key]
+  (.getItem localforage-instance key))
+
+(defn get-keys
+  []
+  (.keys localforage-instance))
+
+(defn get-nfs-dbs
+  []
+  (p/let [ks (get-keys)]
+    (->> (filter (fn [k] (string/starts-with? k (str config/idb-db-prefix config/local-db-prefix))) ks)
+         (map #(string/replace-first % config/idb-db-prefix "")))))

src/main/frontend/mixins.cljs

@@ -61,16 +61,16 @@
                  ;; If the click target is outside of current node
                   (when-not (dom/contains dom-node (.. e -target))
                     (on-hide state e :click))))
-        (when visibilitychange?
-          (listen state js/window "visibilitychange"
-                  (fn [e]
-                    (on-hide state e :visibilitychange))))
-        (listen state dom-node "keydown"
+        (listen state js/window "keydown"
                 (fn [e]
                   (case (.-keyCode e)
                     ;; Esc
                     27 (on-hide state e :esc)
-                    nil)))))
+                    nil)))
+        (when visibilitychange?
+          (listen state js/window "visibilitychange"
+                  (fn [e]
+                    (on-hide state e :visibilitychange))))))
     (catch js/Error e
       ;; TODO: Unable to find node on an unmounted component.
       nil)))
@@ -130,11 +130,7 @@
      :did-remount (fn [old-state new-state]
                     (detach old-state)
                     (attach-listeners new-state)
-                    new-state)
-     ;; :will-unmount (fn [state]
-     ;;                 (detach state)
-     ;;                 state)
-})))
+                    new-state)})))
 
 (defn modal
   [k]

src/main/frontend/state.cljs

@@ -16,7 +16,6 @@
   (atom
    {:route-match nil
     :today nil
-    :daily/migrating? nil
     :db/batch-txs (async/chan 100)
     :notification/show? false
     :notification/content nil
@@ -25,14 +24,10 @@
     :repo/importing-to-db? nil
     :repo/sync-status {}
     :repo/changed-files nil
+    :nfs/loading-files? nil
     ;; TODO: how to detect the network reliably?
     :network/online? true
     :indexeddb/support? true
-    ;; TODO: save in local storage so that if :changed? is true when user
-    ;; reloads the browser, the app should re-index the repo (another way
-    ;; is to save all the tx data since :last-stored-at)
-    ;; repo -> {:last-stored-at :last-modified-at}
-    :repo/persist-status {}
     :me nil
     :git/current-repo (storage/get :git/current-repo)
     :git/status {}
@@ -78,6 +73,12 @@
     :editor/block nil
     :editor/block-dom-id nil
     :editor/set-timestamp-block nil
+    :editor/last-input-time nil
+    :db/last-transact-time {}
+    :db/last-persist-transact-ids {}
+    ;; whether database is persisted
+    :db/persisted? {}
+    :db/latest-txs (or (storage/get-transit :db/latest-txs) {})
     :cursor-range nil
 
     :selection/mode false
@@ -93,7 +94,8 @@
     :preferred-language (storage/get :preferred-language)
 
     ;; all notification contents as k-v pairs
-    :notification/contents {}}))
+    :notification/contents {}
+    :graph/syncing? false}))
 
 (defn get-route-match
   []
@@ -194,8 +196,10 @@
 
 (defn get-pages-directory
   []
-  (when-let [repo (get-current-repo)]
-    (:pages-directory (get-config repo))))
+  (or
+   (when-let [repo (get-current-repo)]
+     (:pages-directory (get-config repo)))
+   "pages"))
 
 (defn org-mode-file-link?
   [repo]
@@ -237,6 +241,18 @@
   []
   (get-in @state [:me :repos]))
 
+(defn set-repos!
+  [repos]
+  (set-state! [:me :repos] repos))
+
+(defn add-repo!
+  [repo]
+  (when repo
+    (update-state! [:me :repos]
+                   (fn [repos]
+                     (->> (conj repos repo)
+                          (distinct))))))
+
 (defn set-current-repo!
   [repo]
   (swap! state assoc :git/current-repo repo)
@@ -778,18 +794,6 @@
          :modal/show? false
          :modal/panel-content nil))
 
-(defn update-repo-last-stored-at!
-  [repo]
-  (swap! state assoc-in [:repo/persist-status repo :last-stored-at] (util/time-ms)))
-
-(defn get-repo-persist-status
-  []
-  (:repo/persist-status @state))
-
-(defn mark-repo-as-changed!
-  [repo _tx-id]
-  (swap! state assoc-in [:repo/persist-status repo :last-modified-at] (util/time-ms)))
-
 (defn get-db-batch-txs-chan
   []
   (:db/batch-txs @state))
@@ -801,13 +805,6 @@
     (when-let [chan (get-db-batch-txs-chan)]
       (async/put! chan f))))
 
-(defn repos-need-to-be-stored?
-  []
-  (let [status (vals (get-repo-persist-status))]
-    (some (fn [{:keys [last-stored-at last-modified-at]}]
-            (> last-modified-at last-stored-at))
-          status)))
-
 (defn get-left-sidebar-open?
   []
   (get-in @state [:ui/left-sidebar-open?]))
@@ -816,10 +813,6 @@
   [value]
   (set-state! :ui/left-sidebar-open? value))
 
-(defn set-daily-migrating!
-  [value]
-  (set-state! :daily/migrating? value))
-
 (defn set-developer-mode!
   [value]
   (set-state! :ui/developer-mode? value)
@@ -886,7 +879,80 @@
   []
   (:commands (get-config)))
 
+(defn set-graph-syncing?
+  [value]
+  (set-state! :graph/syncing? value))
+
+(defn set-loading-files!
+  [value]
+  (set-state! :repo/loading-files? value))
+
+(defn set-importing-to-db!
+  [value]
+  (set-state! :repo/importing-to-db? value))
+
+(defn set-editor-last-input-time!
+  [repo time]
+  (swap! state assoc-in [:editor/last-input-time repo] time))
+
+(defn set-last-transact-time!
+  [repo time]
+  (swap! state assoc-in [:db/last-transact-time repo] time)
+
+  ;; THINK: new block, indent/outdent, drag && drop, etc.
+  (set-editor-last-input-time! repo time))
+
+(defn set-db-persisted!
+  [repo value]
+  (swap! state assoc-in [:db/persisted? repo] value))
+
+(defn db-idle?
+  [repo]
+  (when repo
+    (when-let [last-time (get-in @state [:db/last-transact-time repo])]
+      (let [now (util/time-ms)]
+        (>= (- now last-time) 3000)))))
+
+(defn input-idle?
+  [repo]
+  (when repo
+    (or
+     (when-let [last-time (get-in @state [:editor/last-input-time repo])]
+       (let [now (util/time-ms)]
+         (>= (- now last-time) 3000)))
+     ;; not in editing mode
+     (not (get-edit-input-id)))))
+
+(defn set-last-persist-transact-id!
+  [repo files? id]
+  (swap! state assoc-in [:db/last-persist-transact-ids :repo files?] id))
+
+(defn get-last-persist-transact-id
+  [repo files?]
+  (get-in @state [:db/last-persist-transact-ids :repo files?]))
+
+(defn persist-transaction!
+  [repo files? tx-id tx-data]
+  (when (seq tx-data)
+    (let [latest-txs (:db/latest-txs @state)
+          last-persist-tx-id (get-last-persist-transact-id repo files?)
+          latest-txs (if last-persist-tx-id
+                      (update-in latest-txs [repo files?]
+                                 (fn [result]
+                                   (remove (fn [tx] (<= (:tx-id tx) last-persist-tx-id)) result)))
+                      latest-txs)
+         new-txs (update-in latest-txs [repo files?] (fn [result]
+                                                       (vec (conj result {:tx-id tx-id
+                                                                          :tx-data tx-data}))))]
+     (storage/set-transit! :db/latest-txs new-txs)
+     (set-state! :db/latest-txs new-txs))))
+
+(defn get-repo-latest-txs
+  [repo file?]
+  (get-in (:db/latest-txs @state) [repo file?]))
+
 ;; TODO: Move those to the uni `state`
+
 (defonce editor-op (atom nil))
 (defn set-editor-op!
   [value]

src/main/frontend/storage.cljs

@@ -1,8 +1,8 @@
 (ns frontend.storage
   (:refer-clojure :exclude [get set remove])
-  (:require [cljs.reader :as reader]))
+  (:require [cljs.reader :as reader]
+            [datascript.transit :as dt]))
 
-;; TODO: deprecate this, will persistent datascript
 (defn get
   [key]
   (reader/read-string ^js (.getItem js/localStorage (name key))))
@@ -11,6 +11,14 @@
   [key value]
   (.setItem ^js js/localStorage (name key) (pr-str value)))
 
+(defn get-transit
+  [key]
+  (dt/read-transit-str ^js (.getItem js/localStorage (name key))))
+
+(defn set-transit!
+  [key value]
+  (.setItem ^js js/localStorage (name key) (dt/write-transit-str value)))
+
 (defn get-json
   [key]
   (when-let [value (.getItem js/localStorage (name key))]

src/main/frontend/ui.cljs

@@ -384,11 +384,16 @@
       :aria-hidden "true"}]]])
 
 (defn tooltip
-  [label children]
-  [:div.Tooltip {:style {:display "inline"}}
-   [:div {:class "Tooltip__label"}
-    label]
-   children])
+  ([label children]
+   (tooltip label children {}))
+  ([label children {:keys [label-style]}]
+   [:div.Tooltip {:style {:display "inline"}}
+    [:div (cond->
+            {:class "Tooltip__label"}
+            label-style
+            (assoc :style label-style))
+     label]
+    children]))
 
 (defonce modal-show? (atom false))
 (rum/defc modal-overlay

src/main/frontend/util.cljs

@@ -19,6 +19,11 @@
             [clojure.pprint :refer [pprint]]
             [goog.userAgent]))
 
+(extend-protocol IPrintWithWriter
+  js/Symbol
+  (-pr-writer [sym writer _]
+    (-write writer (str "\"" (.toString sym) "\""))))
+
 ;; envs
 (defn ios?
   []
@@ -946,6 +951,14 @@
   [file]
   (last (string/split file #"\.")))
 
+(defn get-dir-and-basename
+  [path]
+  (let [parts (string/split path "/")
+        basename (last parts)
+        dir (->> (butlast parts)
+                 (string/join "/"))]
+    [dir basename]))
+
 (defn get-relative-path
   [current-file-path another-file-path]
   (let [directories-f #(butlast (string/split % "/"))

src/main/frontend/utils.js

@@ -74,7 +74,83 @@ export var getSelectionText = function () {
     }
   }
 
-  return ''
+  return '';
+}
+
+// Modified from https://github.com/GoogleChromeLabs/browser-nativefs
+// because shadow-cljs doesn't handle this babel transform
+export var getFiles = async function (dirHandle, recursive, cb, path = dirHandle.name) {
+  const dirs = [];
+  const files = [];
+  for await (const entry of dirHandle.values()) {
+    const nestedPath = `${path}/${entry.name}`;
+    if (entry.kind === 'file') {
+      cb(nestedPath, entry);
+      files.push(
+        entry.getFile().then((file) => {
+          Object.defineProperty(file, 'webkitRelativePath', {
+            configurable: true,
+            enumerable: true,
+            get: () => nestedPath,
+          });
+          Object.defineProperty(file, 'handle', {
+            configurable: true,
+            enumerable: true,
+            get: () => entry,
+          });
+          return file;
+        }
+        )
+      );
+    } else if (entry.kind === 'directory' && recursive) {
+      cb(nestedPath, entry);
+      dirs.push(getFiles(entry, recursive, cb, nestedPath));
+    }
+  }
+
+  return [(await Promise.all(dirs)), (await Promise.all(files))];
+};
+
+export var verifyPermission = async function (handle, readWrite) {
+  const options = {};
+  if (readWrite) {
+    options.mode = 'readwrite';
+  }
+  // Check if permission was already granted. If so, return true.
+  if ((await handle.queryPermission(options)) === 'granted') {
+    return true;
+  }
+  // Request permission. If the user grants permission, return true.
+  if ((await handle.requestPermission(options)) === 'granted') {
+    return true;
+  }
+  // The user didn't grant permission, so return false.
+  return false;
+}
+
+export var openDirectory = async function (options = {}, cb) {
+  options.recursive = options.recursive || false;
+  const handle = await window.showDirectoryPicker({ mode: 'readwrite' });
+  const _ask = await verifyPermission(handle, true);
+  return [handle, getFiles(handle, options.recursive, cb)];
+};
+
+export var writeFile = async function (fileHandle, contents) {
+  // Create a FileSystemWritableFileStream to write to.
+  const writable = await fileHandle.createWritable();
+  // Write the contents of the file to the stream.
+  await writable.write(contents);
+  // Close the file and write the contents to disk.
+  await writable.close();
+};
+
+export var nfsSupported = function () {
+  if ('chooseFileSystemEntries' in self) {
+    return 'chooseFileSystemEntries';
+  } else if ('showOpenFilePicker' in self) {
+    return 'showOpenFilePicker';
+  }
+  return false;
 }
 
 const inputTypes = [

start-windows.bat

@@ -1,14 +0,0 @@
-@echo off
-SET ENVIRONMENT=dev
-SET JWT_SECRET=4fa183cf1d28460498b13330835e80ad
-SET COOKIE_SECRET=10a42ca724e34f4db6086a772d787034
-SET DATABASE_URL=postgres://localhost:5432/logseq
-SET GITHUB_APP2_ID=78728
-SET GITHUB_APP2_KEY=xxxxxxxxxxxxxxxxxxxx
-SET GITHUB_APP2_SECRET=xxxxxxxxxxxxxxxxxxxx
-SET GITHUB_APP_PEM=
-SET LOG_PATH=%AppData%\..\Local\Temp\logseq
-
-pg_ctl start
-start cmd.exe /k "java -Duser.timezone=UTC -jar logseq.jar"
-yarn && yarn watch

yarn.lock

@@ -963,6 +963,13 @@ create-hmac@^1.1.0, create-hmac@^1.1.2, create-hmac@^1.1.4:
     safe-buffer "^5.0.1"
     sha.js "^2.4.8"
 
+cross-env@^7.0.3:
+  version "7.0.3"
+  resolved "https://registry.yarnpkg.com/cross-env/-/cross-env-7.0.3.tgz#865264b29677dc015ba8418918965dd232fc54cf"
+  integrity sha512-+/HKd6EgcQCJGh2PSjZuUitQBQynKor4wrFbRg4DtAgS1aWO+gU52xpH7M9ScGgXSYmAVS9bIJ8EzuaGw0oNAw==
+  dependencies:
+    cross-spawn "^7.0.1"
+
 cross-spawn@^6.0.0, cross-spawn@^6.0.5:
   version "6.0.5"
   resolved "https://registry.yarnpkg.com/cross-spawn/-/cross-spawn-6.0.5.tgz#4a5ec7c64dfae22c3a14124dbacdee846d80cbc4"
@@ -974,6 +981,15 @@ cross-spawn@^6.0.0, cross-spawn@^6.0.5:
     shebang-command "^1.2.0"
     which "^1.2.9"
 
+cross-spawn@^7.0.1:
+  version "7.0.3"
+  resolved "https://registry.yarnpkg.com/cross-spawn/-/cross-spawn-7.0.3.tgz#f73a85b9d5d41d045551c177e2882d4ac85728a6"
+  integrity sha512-iRDPJKUPVEND7dHPO8rkbOnPpyDygcDFtWjpeWNCgy8WP2rXcxXL8TskReQl6OrB2G7+UJrags1q15Fudc7G6w==
+  dependencies:
+    path-key "^3.1.0"
+    shebang-command "^2.0.0"
+    which "^2.0.1"
+
 crypto-browserify@^3.11.0:
   version "3.12.0"
   resolved "https://registry.yarnpkg.com/crypto-browserify/-/crypto-browserify-3.12.0.tgz#396cf9f3137f03e4b8e532c58f698254e00f80ec"
@@ -2131,7 +2147,7 @@ ignore@^4.0.3:
   resolved "https://registry.yarnpkg.com/ignore/-/ignore-4.0.6.tgz#750e3db5862087b4737ebac8207ffd1ef27b25fc"
   integrity sha512-cyFDKrqc/YdcWFniJhzI42+AzS+gNwmUzOSFcRCQYwySuBBBy/KjuxWLZ/FHEH6Moq1NizMOBWyTcv8O4OZIMg==
 
-ignore@^5.1.4:
+ignore@^5.1.4, ignore@^5.1.8:
   version "5.1.8"
   resolved "https://registry.yarnpkg.com/ignore/-/ignore-5.1.8.tgz#f150a8b50a34289b33e22f5889abd4d8016f0e57"
   integrity sha512-BMpfD7PpiETpBl/A6S498BaIJ6Y/ABT93ETbby2fP00v4EbvPBXWEoaR1UBPKs3iR53pJY7EtZk5KACI57i1Uw==
@@ -3364,6 +3380,11 @@ path-key@^2.0.0, path-key@^2.0.1:
   resolved "https://registry.yarnpkg.com/path-key/-/path-key-2.0.1.tgz#411cadb574c5a140d3a4b1910d40d80cc9f40b40"
   integrity sha1-QRyttXTFoUDTpLGRDUDYDMn0C0A=
 
+path-key@^3.1.0:
+  version "3.1.1"
+  resolved "https://registry.yarnpkg.com/path-key/-/path-key-3.1.1.tgz#581f6ade658cbba65a0d3380de7753295054f375"
+  integrity sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==
+
 path-parse@^1.0.6:
   version "1.0.6"
   resolved "https://registry.yarnpkg.com/path-parse/-/path-parse-1.0.6.tgz#d62dbb5679405d72c4737ec58600e9ddcf06d24c"
@@ -4372,11 +4393,23 @@ shebang-command@^1.2.0:
   dependencies:
     shebang-regex "^1.0.0"
 
+shebang-command@^2.0.0:
+  version "2.0.0"
+  resolved "https://registry.yarnpkg.com/shebang-command/-/shebang-command-2.0.0.tgz#ccd0af4f8835fbdc265b82461aaf0c36663f34ea"
+  integrity sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==
+  dependencies:
+    shebang-regex "^3.0.0"
+
 shebang-regex@^1.0.0:
   version "1.0.0"
   resolved "https://registry.yarnpkg.com/shebang-regex/-/shebang-regex-1.0.0.tgz#da42f49740c0b42db2ca9728571cb190c98efea3"
   integrity sha1-2kL0l0DAtC2yypcoVxyxkMmO/qM=
 
+shebang-regex@^3.0.0:
+  version "3.0.0"
+  resolved "https://registry.yarnpkg.com/shebang-regex/-/shebang-regex-3.0.0.tgz#ae16f1644d873ecad843b0307b143362d4c42172"
+  integrity sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==
+
 shell-quote@^1.6.1:
   version "1.7.2"
   resolved "https://registry.yarnpkg.com/shell-quote/-/shell-quote-1.7.2.tgz#67a7d02c76c9da24f99d20808fcaded0e0e04be2"
@@ -5119,6 +5152,13 @@ which@^1.2.14, which@^1.2.9, which@^1.3.1:
   dependencies:
     isexe "^2.0.0"
 
+which@^2.0.1:
+  version "2.0.2"
+  resolved "https://registry.yarnpkg.com/which/-/which-2.0.2.tgz#7c6a8dd0a636a0327e10b59c9286eee93f3f51b1"
+  integrity sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==
+  dependencies:
+    isexe "^2.0.0"
+
 wrap-ansi@^2.0.0:
   version "2.1.0"
   resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-2.1.0.tgz#d8fc3d284dd05794fe84973caecdd1cf824fdd85"