Apache VFS 移動(dòng)FTP文件太慢的原因

項(xiàng)目中使用VFS移動(dòng)文件是通過使用FileSystemManager的resolveFile方法獲得FileObject，然后調(diào)用其moveTo方法來達(dá)到FTP文件移動(dòng)的目的。

我們使用的FileSystemManager是默認(rèn)的DefaultFileSystemManager，在操作FTP文件的時(shí)候會(huì)調(diào)用AbstractOriginatingFileProvider的findFile方法。

protected FileObject findFile(final FileName name, final FileSystemOptions fileSystemOptions)
            throws FileSystemException {
        // Check in the cache for the file system
        final FileName rootName = getContext().getFileSystemManager().resolveName(name, FileName.ROOT_PATH);

        final FileSystem fs = getFileSystem(rootName, fileSystemOptions);

        // Locate the file
        // return fs.resolveFile(name.getPath());
        return fs.resolveFile(name);
    }

從這里可以看到會(huì)使用FileName獲取到一個(gè)FileSystem，然后調(diào)用FlieSystem的resolveFile方法。這個(gè)FileName是從FTP的uri中解析出來的。FTP的uri（例如：ftp://username:password@host:port/）如果username，password，host，port相同，這里取到的FileSystem是同一個(gè)。這里涉及到兩個(gè)重要的類。

  1. FileObject，這里是FtpFileObject
protected FtpFileObject(final AbstractFileName name, final FtpFileSystem fileSystem, final FileName rootName)
            throws FileSystemException {
        super(name, fileSystem);
        final String relPath = UriParser.decode(rootName.getRelativeName(name));
        if (".".equals(relPath)) {
            // do not use the "." as path against the ftp-server
            // e.g. the uu.net ftp-server do a recursive listing then
            // this.relPath = UriParser.decode(rootName.getPath());
            // this.relPath = ".";
            this.relPath = null;
        } else {
            this.relPath = relPath;
        }
    }

從構(gòu)造函數(shù)可以看出，并沒有做太多事情，而且最關(guān)鍵的屬性

private FTPFile fileInfo;

沒有初始化。

  1. FileSystem，這里是FtpFileSystem
public void putClient(final FtpClient client) {
        // Save client for reuse if none is idle.
        if (!idleClient.compareAndSet(null, client)) {
            // An idle client is already present so close the connection.
            closeConnection(client);
        }
    }
public FtpClient getClient() throws FileSystemException {
        FtpClient client = idleClient.getAndSet(null);

        if (client == null || !client.isConnected()) {
            client = createWrapper();
        }

        return client;
    }

這個(gè)類就是對(duì)FtpClient進(jìn)行了封裝，操作FTP文件時(shí)會(huì)先調(diào)用getClient()，操作完成后再調(diào)用putClient。這個(gè)類使用AtomicReference來保持他只持有一個(gè)FtpClient，每次get的時(shí)候會(huì)置null，如果有其他的線程get，那么會(huì)創(chuàng)建一個(gè)新的client返回。在put的時(shí)候，如果這個(gè)類已經(jīng)持有一個(gè)client了，就把put進(jìn)來的client關(guān)掉。

接下來看AbstractFileObject的moveTo方法

@Override
    public void moveTo(final FileObject destFile) throws FileSystemException {
        if (canRenameTo(destFile)) {
            if (!getParent().isWriteable()) {
                throw new FileSystemException("vfs.provider/rename-parent-read-only.error", getName(),
                        getParent().getName());
            }
        } else {
            if (!isWriteable()) {
                throw new FileSystemException("vfs.provider/rename-read-only.error", getName());
            }
        }

        if (destFile.exists() && !isSameFile(destFile)) {
            destFile.deleteAll();
            // throw new FileSystemException("vfs.provider/rename-dest-exists.error", destFile.getName());
        }

        if (canRenameTo(destFile)) {
            // issue rename on same filesystem
            try {
                attach();
                // remember type to avoid attach
                final FileType srcType = getType();

                doRename(destFile);

                FileObjectUtils.getAbstractFileObject(destFile).handleCreate(srcType);
                destFile.close(); // now the destFile is no longer imaginary. force reattach.

                handleDelete(); // fire delete-events. This file-object (src) is like deleted.
            } catch (final RuntimeException re) {
                throw re;
            } catch (final Exception exc) {
                throw new FileSystemException("vfs.provider/rename.error", exc, getName(), destFile.getName());
            }
        } else {
            // different fs - do the copy/delete stuff

            destFile.copyFrom(this, Selectors.SELECT_SELF);

            if ((destFile.getType().hasContent()
                    && destFile.getFileSystem().hasCapability(Capability.SET_LAST_MODIFIED_FILE)
                    || destFile.getType().hasChildren()
                            && destFile.getFileSystem().hasCapability(Capability.SET_LAST_MODIFIED_FOLDER))
                    && fs.hasCapability(Capability.GET_LAST_MODIFIED)) {
                destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
            }

            deleteSelf();
        }

    }

這個(gè)方法也不復(fù)雜，移動(dòng)文件有兩種情況

  1. 源文件和目標(biāo)文件在同一個(gè)filesystem，使用doRename
  2. 源文件和目標(biāo)文件不在同一個(gè)filesystem，使用copyFrom

前面我們已經(jīng)知道username，password，host，port相同的時(shí)候取到的就是同一個(gè)filesystem，所以這里判斷源文件和目標(biāo)文件是否在同一個(gè)filesystem也很簡(jiǎn)單，直接用==判斷。

 @Override
    public boolean canRenameTo(final FileObject newfile) {
        return fs == newfile.getFileSystem();
    }

doRename的實(shí)現(xiàn)原理就是調(diào)用FTPClient的rename方法，而這個(gè)FTPClient是通過FTP的RNFR和RNTO指令實(shí)現(xiàn)的。
copyFrom則是通過FTP協(xié)議中的RETR和STOR命令來下載上傳實(shí)現(xiàn)的。

目前來看，文件移動(dòng)都沒什么問題，然而項(xiàng)目中導(dǎo)致移動(dòng)文件慢的竟然是這個(gè)方法

@Override
    public boolean exists() throws FileSystemException {
        return getType() != FileType.IMAGINARY;
    }

不管源文件與目標(biāo)文件是否在同一個(gè)文件系統(tǒng)都會(huì)對(duì)源文件和目標(biāo)文件執(zhí)行這個(gè)getType()方法。這個(gè)方法最終會(huì)調(diào)用FtpFileObject的doGetType()方法。

@Override
    protected FileType doGetType() throws Exception {
        // VFS-210
        synchronized (getFileSystem()) {
            if (this.fileInfo == null) {
                getInfo(false);
            }

            if (this.fileInfo == UNKNOWN) {
                return FileType.IMAGINARY;
            } else if (this.fileInfo.isDirectory()) {
                return FileType.FOLDER;
            } else if (this.fileInfo.isFile()) {
                return FileType.FILE;
            } else if (this.fileInfo.isSymbolicLink()) {
                final FileObject linkDest = getLinkDestination();
                // VFS-437: We need to check if the symbolic link links back to the symbolic link itself
                if (this.isCircular(linkDest)) {
                    // If the symbolic link links back to itself, treat it as an imaginary file to prevent following
                    // this link. If the user tries to access the link as a file or directory, the user will end up with
                    // a FileSystemException warning that the file cannot be accessed. This is to prevent the infinite
                    // call back to doGetType() to prevent the StackOverFlow
                    return FileType.IMAGINARY;
                }
                return linkDest.getType();

            }
        }
        throw new FileSystemException("vfs.provider.ftp/get-type.error", getName());
    }

上面已經(jīng)說過，F(xiàn)tpFileObject的fileInfo沒有初始化，所以這里會(huì)執(zhí)行g(shù)etInfo方法，而getInfo方法又會(huì)調(diào)用getChildFile

private FTPFile getChildFile(final String name, final boolean flush) throws IOException {
        /*
         * If we should flush cached children, clear our children map unless we're in the middle of a refresh in which
         * case we've just recently refreshed our children. No need to do it again when our children are refresh()ed,
         * calling getChildFile() for themselves from within getInfo(). See getChildren().
         */
        if (flush && !inRefresh) {
            children = null;
        }

        // List the children of this file
        doGetChildren();

        // VFS-210
        if (children == null) {
            return null;
        }

        // Look for the requested child
        final FTPFile ftpFile = children.get(name);
        return ftpFile;
    }

就是這里，我們可以看到，獲取某個(gè)文件時(shí)，會(huì)先獲取父路徑的所有子文件，然后從子文件中獲取你要的那個(gè)文件。
如果你要的那個(gè)文件在一個(gè)文件非常多的目錄里，而且關(guān)閉了緩存，你每獲取這個(gè)目錄的一個(gè)文件就要把目錄里的所有文件列一次。

FTPClient是可以通過listFiles列出單個(gè)文件的，所以解決辦法就是

  1. 使用緩存
  2. 不要用VFS了，直接用FTPClient的rename方法（僅限于同一個(gè)FTPClient，如果時(shí)跨文件服務(wù)器的需要FTPClient的上傳下載實(shí)現(xiàn)）。
    下面附上解決辦法2的代碼。
    代碼很簡(jiǎn)單，大多數(shù)都是解析URI的，全塞一個(gè)類里了，如果真要用建議把一些代碼拆出來。
import java.io.IOException;
import java.util.Arrays;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;

import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.vfs2.FileSystemException;
import org.apache.commons.vfs2.FileSystemOptions;
import org.apache.commons.vfs2.provider.UriParser;
import org.apache.commons.vfs2.provider.ftp.FtpClientFactory;
import org.apache.commons.vfs2.provider.ftp.FtpFileSystemConfigBuilder;
import org.apache.commons.vfs2.util.Cryptor;
import org.apache.commons.vfs2.util.CryptorFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.google.common.collect.Maps;

public class FtpUtil {
    
    private static Logger logger = LoggerFactory.getLogger(FtpUtil.class);
    
    private final static Map<Auth, AtomicReference<FTPClient>> clients = Maps.newConcurrentMap();
    
    public static boolean move(String src, String tar) throws IOException {
        FtpPath srcFtpPath = parse(src);
        FtpPath tarFtpPath = parse(tar);
        if (!srcFtpPath.auth.equals(tarFtpPath.auth)) {
            throw new UnsupportedOperationException("源目錄和目標(biāo)目錄的ftp服務(wù)器連接信息不一致");
        }
        FTPClient ftpClient = getFTPClient(srcFtpPath.auth);
        try {
            return ftpClient.rename(srcFtpPath.path, tarFtpPath.path);
        } catch (IOException e) {
            closeConnection(ftpClient);
            throw e;
        } finally {
            putFTPClient(srcFtpPath.auth, ftpClient);
        }
    }
    
    public static FtpPath parse(String uri) throws FileSystemException {
        FtpPath ftpPath = new FtpPath();
        StringBuilder name = new StringBuilder();
        UriParser.extractScheme(uri, name);
        // Expecting "http://"
        if (name.length() < 2 || name.charAt(0) != '/' || name.charAt(1) != '/') {
            throw new FileSystemException("vfs.provider/missing-double-slashes.error", uri);
        }
        name.delete(0, 2);
     // Extract userinfo, and split into username and password
        final String userInfo = extractUserInfo(name);
        final String userName;
        final String password;
        if (userInfo != null) {
            final int idx = userInfo.indexOf(':');
            if (idx == -1) {
                userName = userInfo;
                password = null;
            } else {
                userName = userInfo.substring(0, idx);
                password = userInfo.substring(idx + 1);
            }
        } else {
            userName = null;
            password = null;
        }
        
        String u = UriParser.decode(userName);
        String p = UriParser.decode(password);

        if (p != null && p.startsWith("{") && p.endsWith("}")) {
            try {
                final Cryptor cryptor = CryptorFactory.getCryptor();
                p = cryptor.decrypt(p.substring(1, p.length() - 1));
            } catch (final Exception ex) {
                throw new FileSystemException("Unable to decrypt password", ex);
            }
        }
        
        ftpPath.auth.username = u == null ? null : u.toCharArray();
        ftpPath.auth.password = p == null ? null : p.toCharArray();
        
        // Extract hostname, and normalise (lowercase)
        final String hostName = extractHostName(name);
        if (hostName == null) {
            throw new FileSystemException("vfs.provider/missing-hostname.error", uri);
        }
        ftpPath.auth.host = hostName.toLowerCase();

        // Extract port
        ftpPath.auth.port = extractPort(name, uri);

        // Expecting '/' or empty name
        if (name.length() > 0 && name.charAt(0) != '/') {
            throw new FileSystemException("vfs.provider/missing-hostname-path-sep.error", uri);
        }
        
        ftpPath.path = name.toString();
        return ftpPath;
    }
    
    /**
     * Extracts the user info from a URI.
     *
     * @param name string buffer with the "scheme://" part has been removed already. Will be modified.
     * @return the user information up to the '@' or null.
     */
    private static String extractUserInfo(final StringBuilder name) {
        final int maxlen = name.length();
        for (int pos = 0; pos < maxlen; pos++) {
            final char ch = name.charAt(pos);
            if (ch == '@') {
                // Found the end of the user info
                final String userInfo = name.substring(0, pos);
                name.delete(0, pos + 1);
                return userInfo;
            }
            if (ch == '/' || ch == '?') {
                // Not allowed in user info
                break;
            }
        }

        // Not found
        return null;
    }

    /**
     * Extracts the hostname from a URI.
     *
     * @param name string buffer with the "scheme://[userinfo@]" part has been removed already. Will be modified.
     * @return the host name or null.
     */
    private static String extractHostName(final StringBuilder name) {
        final int maxlen = name.length();
        int pos = 0;
        for (; pos < maxlen; pos++) {
            final char ch = name.charAt(pos);
            if (ch == '/' || ch == ';' || ch == '?' || ch == ':' || ch == '@' || ch == '&' || ch == '=' || ch == '+'
                    || ch == '$' || ch == ',') {
                break;
            }
        }
        if (pos == 0) {
            return null;
        }

        final String hostname = name.substring(0, pos);
        name.delete(0, pos);
        return hostname;
    }

    /**
     * Extracts the port from a URI.
     *
     * @param name string buffer with the "scheme://[userinfo@]hostname" part has been removed already. Will be
     *            modified.
     * @param uri full URI for error reporting.
     * @return The port, or -1 if the URI does not contain a port.
     * @throws FileSystemException if URI is malformed.
     * @throws NumberFormatException if port number cannot be parsed.
     */
    private static int extractPort(final StringBuilder name, final String uri) throws FileSystemException {
        if (name.length() < 1 || name.charAt(0) != ':') {
            return -1;
        }

        final int maxlen = name.length();
        int pos = 1;
        for (; pos < maxlen; pos++) {
            final char ch = name.charAt(pos);
            if (ch < '0' || ch > '9') {
                break;
            }
        }

        final String port = name.substring(1, pos);
        name.delete(0, pos);
        if (port.length() == 0) {
            throw new FileSystemException("vfs.provider/missing-port.error", uri);
        }

        return Integer.parseInt(port);
    }
    
    private static FTPClient getFTPClient(Auth key) throws IOException {
        AtomicReference<FTPClient> refClient = clients.getOrDefault(key, new AtomicReference<FTPClient>(null));
        
        FTPClient client = refClient.getAndSet(null);
        if (client == null || !client.isConnected()) {
            client = createClient(key);
        }
        return client;
    }
    
    private static FTPClient createClient(Auth key) throws IOException {
        FtpFileSystemConfigBuilder builder = FtpFileSystemConfigBuilder.getInstance();
        FileSystemOptions options = new FileSystemOptions();
        builder.setControlEncoding(options, "UTF-8");
        builder.setServerLanguageCode(options, "zh");
        builder.setPassiveMode(options, true);
        return FtpClientFactory.createConnection(key.host, key.port, key.username, key.password, null, options);
    }
    
    private static void putFTPClient(Auth key, FTPClient client) {
        AtomicReference<FTPClient> refClient = clients.getOrDefault(key, new AtomicReference<FTPClient>(null));
        
        if (!refClient.compareAndSet(null, client)) {
            closeConnection(client);
        }
    }

    private static void closeConnection(FTPClient client) {
        try {
            if (client.isConnected()) {
                client.disconnect();
            }
        } catch (final IOException e) {
            logger.error(e.getMessage(), e);
        }
    }

    private static class Auth {
        String host;
        int port;
        char[] username;
        char[] password;
        
        @Override
        public boolean equals(Object obj) {
            if (this == obj) {
                return true;
            }
            if (obj instanceof Auth) {
                Auth k = (Auth) obj;
                return this.host.equals(k.host) && this.port == k.port && Arrays.equals(this.username, k.username)
                        && Arrays.equals(this.password, k.password);
            }
            return false;
        }
        
        @Override
        public int hashCode() {
            int h = host.hashCode();
            h = 31 * h + port;
            h = 31 * h + username.hashCode();
            h = 31 * h + password.hashCode();
            return h;
        }
    }
    
    private static class FtpPath {
        Auth auth = new Auth();
        String path;
    }
}
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

  • 當(dāng)數(shù)據(jù)量增大到超出了單個(gè)物理計(jì)算機(jī)存儲(chǔ)容量時(shí)，有必要把它分開存儲(chǔ)在多個(gè)不同的計(jì)算機(jī)中。那些管理存儲(chǔ)在多個(gè)網(wǎng)絡(luò)互連的...
    單行線的旋律閱讀 2,083評(píng)論 0贊 7
  • ORA-00001: 違反唯一約束條件 (.) 錯(cuò)誤說明：當(dāng)在唯一索引所對(duì)應(yīng)的列上鍵入重復(fù)值時(shí)，會(huì)觸發(fā)此異常。 O...
    我想起個(gè)好名字閱讀 6,026評(píng)論 0贊 9
  • 2.1 Activity 2.1.1 Activity的生命周期全面分析 典型情況下的生命周期：在用戶參與的情況下...
    AndroidMaster閱讀 3,299評(píng)論 0贊 8
  • Swift1> Swift和OC的區(qū)別1.1> Swift沒有地址/指針的概念1.2> 泛型1.3> 類型嚴(yán)謹(jǐn) 對(duì)...
    cosWriter閱讀 11,689評(píng)論 1贊 32
  • 一、Python簡(jiǎn)介和環(huán)境搭建以及pip的安裝 4課時(shí)實(shí)驗(yàn)課主要內(nèi)容 【Python簡(jiǎn)介】： Python 是一個(gè)...
    _小老虎_閱讀 6,356評(píng)論 0贊 10

友情鏈接更多精彩內(nèi)容