Class FSDirectory
- java.lang.Object
-
- org.apache.lucene.store.Directory
-
- org.apache.lucene.store.FSDirectory
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
- Direct Known Subclasses:
DirectIOLinuxDirectory
,MMapDirectory
,NIOFSDirectory
,SimpleFSDirectory
,WindowsDirectory
public abstract class FSDirectory extends Directory
Base class for Directory implementations that store index files in the file system. There are currently three core subclasses:-
SimpleFSDirectory
is a straightforward implementation using java.io.RandomAccessFile. However, it has poor concurrent performance (multiple threads will bottleneck) as it synchronizes when multiple threads read from the same file. -
NIOFSDirectory
uses java.nio's FileChannel's positional io when reading to avoid synchronization when reading from the same file. Unfortunately, due to a Windows-only Sun JRE bug this is a poor choice for Windows, but on all other platforms this is the preferred choice. Applications usingThread.interrupt()
orFuture.cancel(boolean)
should useSimpleFSDirectory
instead. SeeNIOFSDirectory
java doc for details. -
MMapDirectory
uses memory-mapped IO when reading. This is a good choice if you have plenty of virtual memory relative to your index size, eg if you are running on a 64 bit JRE, or you are running on a 32 bit JRE but your index sizes are small enough to fit into the virtual memory space. Java has currently the limitation of not being able to unmap files from user code. The files are unmapped, when GC releases the byte buffers. Due to this bug in Sun's JRE, MMapDirectory'sIndexInput.close()
is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed. This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of. This class supplies a (possibly dangerous) workaround mentioned in the bug report, which may fail on non-Sun JVMs. Applications usingThread.interrupt()
orFuture.cancel(boolean)
should useSimpleFSDirectory
instead. SeeMMapDirectory
java doc for details.
open(java.io.File)
method, to allow Lucene to choose the best FSDirectory implementation given your environment, and the known limitations of each implementation. For users who have no reason to prefer a specific implementation, it's best to simply useopen(java.io.File)
. For all others, you should instantiate the desired implementation directly.The locking implementation is by default
NativeFSLockFactory
, but can be changed by passing in a customLockFactory
instance.- See Also:
Directory
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected static class
FSDirectory.FSIndexOutput
-
Field Summary
Fields Modifier and Type Field Description static int
DEFAULT_READ_CHUNK_SIZE
Default read chunk size.protected File
directory
protected Set<String>
staleFiles
-
Fields inherited from class org.apache.lucene.store.Directory
isOpen, lockFactory
-
-
Constructor Summary
Constructors Modifier Constructor Description protected
FSDirectory(File path, LockFactory lockFactory)
Create a new FSDirectory for the named location (ctor for subclasses).
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description void
close()
Closes the store to future operations.IndexOutput
createOutput(String name)
Creates an IndexOutput for the file with the given name.void
deleteFile(String name)
Removes an existing file in the directory.protected void
ensureCanWrite(String name)
boolean
fileExists(String name)
Returns true iff a file with the given name exists.long
fileLength(String name)
Returns the length in bytes of a file in the directory.static long
fileModified(File directory, String name)
Returns the time the named file was last modified.long
fileModified(String name)
Returns the time the named file was last modified.protected void
fsync(String name)
File
getDirectory()
File
getFile()
Deprecated.UsegetDirectory()
instead.String
getLockID()
Return a string identifier that uniquely differentiates this Directory instance from other Directory instances.int
getReadChunkSize()
The maximum number of bytes to read at once from the underlying file duringDataInput.readBytes(byte[], int, int)
.String[]
listAll()
Lists all files (not subdirectories) in the directory.static String[]
listAll(File dir)
Lists all files (not subdirectories) in the directory.protected void
onIndexOutputClosed(FSDirectory.FSIndexOutput io)
static FSDirectory
open(File path)
Creates an FSDirectory instance, trying to pick the best implementation given the current environment.static FSDirectory
open(File path, LockFactory lockFactory)
Just likeopen(File)
, but allows you to also specify a customLockFactory
.IndexInput
openInput(String name)
Returns a stream reading an existing file.void
setLockFactory(LockFactory lockFactory)
Set the LockFactory that this Directory instance should use for its locking implementation.void
setReadChunkSize(int chunkSize)
Sets the maximum number of bytes read at once from the underlying file duringDataInput.readBytes(byte[], int, int)
.void
sync(String name)
Deprecated.void
sync(Collection<String> names)
Ensure that any writes to these files are moved to stable storage.String
toString()
For debug output.void
touchFile(String name)
Deprecated.Lucene never uses this API; it will be removed in 4.0.-
Methods inherited from class org.apache.lucene.store.Directory
clearLock, copy, copy, ensureOpen, getLockFactory, makeLock, openInput
-
-
-
-
Field Detail
-
DEFAULT_READ_CHUNK_SIZE
public static final int DEFAULT_READ_CHUNK_SIZE
Default read chunk size. This is a conditional default: on 32bit JVMs, it defaults to 100 MB. On 64bit JVMs, it'sInteger.MAX_VALUE
.- See Also:
setReadChunkSize(int)
-
directory
protected final File directory
-
-
Constructor Detail
-
FSDirectory
protected FSDirectory(File path, LockFactory lockFactory) throws IOException
Create a new FSDirectory for the named location (ctor for subclasses).- Parameters:
path
- the path of the directorylockFactory
- the lock factory to use, or null for the default (NativeFSLockFactory
);- Throws:
IOException
-
-
Method Detail
-
open
public static FSDirectory open(File path) throws IOException
Creates an FSDirectory instance, trying to pick the best implementation given the current environment. The directory returned uses theNativeFSLockFactory
.Currently this returns
MMapDirectory
for most Solaris and Windows 64-bit JREs,NIOFSDirectory
for other non-Windows JREs, andSimpleFSDirectory
for other JREs on Windows. It is highly recommended that you consult the implementation's documentation for your platform before using this method.NOTE: this method may suddenly change which implementation is returned from release to release, in the event that higher performance defaults become possible; if the precise implementation is important to your application, please instantiate it directly, instead. For optimal performance you should consider using
MMapDirectory
on 64 bit JVMs.See above
- Throws:
IOException
-
open
public static FSDirectory open(File path, LockFactory lockFactory) throws IOException
Just likeopen(File)
, but allows you to also specify a customLockFactory
.- Throws:
IOException
-
setLockFactory
public void setLockFactory(LockFactory lockFactory) throws IOException
Description copied from class:Directory
Set the LockFactory that this Directory instance should use for its locking implementation. Each * instance of LockFactory should only be used for one directory (ie, do not share a single instance across multiple Directories).- Overrides:
setLockFactory
in classDirectory
- Parameters:
lockFactory
- instance ofLockFactory
.- Throws:
IOException
-
listAll
public static String[] listAll(File dir) throws IOException
Lists all files (not subdirectories) in the directory. This method never returns null (throwsIOException
instead).- Throws:
NoSuchDirectoryException
- if the directory does not exist, or does exist but is not a directory.IOException
- if list() returns null
-
listAll
public String[] listAll() throws IOException
Lists all files (not subdirectories) in the directory.- Specified by:
listAll
in classDirectory
- Throws:
NoSuchDirectoryException
- if the directory is not prepared for any write operations (such asDirectory.createOutput(String)
).IOException
- in case of other IO errors- See Also:
listAll(File)
-
fileExists
public boolean fileExists(String name)
Returns true iff a file with the given name exists.- Specified by:
fileExists
in classDirectory
-
fileModified
public long fileModified(String name)
Returns the time the named file was last modified.- Specified by:
fileModified
in classDirectory
-
fileModified
public static long fileModified(File directory, String name)
Returns the time the named file was last modified.
-
touchFile
@Deprecated public void touchFile(String name)
Deprecated.Lucene never uses this API; it will be removed in 4.0.Set the modified time of an existing file to now.
-
fileLength
public long fileLength(String name) throws IOException
Returns the length in bytes of a file in the directory.- Specified by:
fileLength
in classDirectory
- Parameters:
name
- the name of the file for which to return the length.- Throws:
FileNotFoundException
- if the file does not exist.IOException
- if there was an IO error while retrieving the file's length.
-
deleteFile
public void deleteFile(String name) throws IOException
Removes an existing file in the directory.- Specified by:
deleteFile
in classDirectory
- Throws:
IOException
-
createOutput
public IndexOutput createOutput(String name) throws IOException
Creates an IndexOutput for the file with the given name.- Specified by:
createOutput
in classDirectory
- Throws:
IOException
-
ensureCanWrite
protected void ensureCanWrite(String name) throws IOException
- Throws:
IOException
-
onIndexOutputClosed
protected void onIndexOutputClosed(FSDirectory.FSIndexOutput io)
-
sync
@Deprecated public void sync(String name) throws IOException
Deprecated.Description copied from class:Directory
Ensure that any writes to this file are moved to stable storage. Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index.- Overrides:
sync
in classDirectory
- Throws:
IOException
-
sync
public void sync(Collection<String> names) throws IOException
Description copied from class:Directory
Ensure that any writes to these files are moved to stable storage. Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index.
NOTE: Clients may call this method for same files over and over again, so some impls might optimize for that. For other impls the operation can be a noop, for various reasons.- Overrides:
sync
in classDirectory
- Throws:
IOException
-
openInput
public IndexInput openInput(String name) throws IOException
Description copied from class:Directory
Returns a stream reading an existing file.- Specified by:
openInput
in classDirectory
- Throws:
IOException
-
getLockID
public String getLockID()
Description copied from class:Directory
Return a string identifier that uniquely differentiates this Directory instance from other Directory instances. This ID should be the same if two Directory instances (even in different JVMs and/or on different machines) are considered "the same index". This is how locking "scopes" to the right index.
-
close
public void close()
Closes the store to future operations.
-
getFile
@Deprecated public File getFile()
Deprecated.UsegetDirectory()
instead.
-
getDirectory
public File getDirectory()
- Returns:
- the underlying filesystem directory
-
setReadChunkSize
public final void setReadChunkSize(int chunkSize)
Sets the maximum number of bytes read at once from the underlying file duringDataInput.readBytes(byte[], int, int)
. The default value isDEFAULT_READ_CHUNK_SIZE
;This was introduced due to Sun JVM Bug 6478546, which throws an incorrect OutOfMemoryError when attempting to read too many bytes at once. It only happens on 32bit JVMs with a large maximum heap size.
Changes to this value will not impact any already-opened
IndexInput
s. You should call this before attempting to open an index on the directory.NOTE: This value should be as large as possible to reduce any possible performance impact. If you still encounter an incorrect OutOfMemoryError, trying lowering the chunk size.
-
getReadChunkSize
public final int getReadChunkSize()
The maximum number of bytes to read at once from the underlying file duringDataInput.readBytes(byte[], int, int)
.- See Also:
setReadChunkSize(int)
-
fsync
protected void fsync(String name) throws IOException
- Throws:
IOException
-
-